[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:27:35 dist-test-slave-1xrh krb5kdc[27662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:27:35 dist-test-slave-1xrh krb5kdc[27662](info): set up 2 sockets
Aug 07 18:27:35 dist-test-slave-1xrh krb5kdc[27662](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 18:27:36.225065 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:27:36 dist-test-slave-1xrh krb5kdc[27662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:27:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:27:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:27:36.424432 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:39099
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:39099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:36.765740 27675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:36.766088 27675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:36.766260 27675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:36.773572 27675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:27:36.773701 27675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:36.773818 27675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:36.773924 27675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:27:36.774013 27675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:27:36.784044 27675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:39099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:39099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:36.785494 27675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:36.786968 27675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:36.806689 27681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:36.806739 27680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:36.806761 27683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:36.808259 27675 server_base.cc:1034] running on GCE node
I20260807 18:27:36.809213 27675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:36.810724 27675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:36.811930 27675 hybrid_clock.cc:648] HybridClock initialized: now 1786127256811861 us; error 83 us; skew 500 ppm
I20260807 18:27:36.812278 27675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:36 dist-test-slave-1xrh krb5kdc[27662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127256, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:36.821237 27675 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:27:36.822945 27675 webserver.cc:460] Webserver started at http://127.27.1.254:41165/ using document root <none> and password file <none>
I20260807 18:27:36.823750 27675 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:36.823869 27675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:36.824136 27675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:36.827538 27675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "5c236e2df27a401aa2e12764dd1cce76"
format_stamp: "Formatted at 2026-08-07 18:27:36 on dist-test-slave-1xrh"
server_key: "9e5216c260f1698fc451555009259b84"
server_key_iv: "0468448448b368b65b43d750a8a4d9d5"
server_key_version: "encryptionkey@0"
I20260807 18:27:36.828622 27675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "5c236e2df27a401aa2e12764dd1cce76"
format_stamp: "Formatted at 2026-08-07 18:27:36 on dist-test-slave-1xrh"
server_key: "9e5216c260f1698fc451555009259b84"
server_key_iv: "0468448448b368b65b43d750a8a4d9d5"
server_key_version: "encryptionkey@0"
I20260807 18:27:36.834175 27675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:27:36.837651 27690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:36.838917 27675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 18:27:36.839125 27675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "5c236e2df27a401aa2e12764dd1cce76"
format_stamp: "Formatted at 2026-08-07 18:27:36 on dist-test-slave-1xrh"
server_key: "9e5216c260f1698fc451555009259b84"
server_key_iv: "0468448448b368b65b43d750a8a4d9d5"
server_key_version: "encryptionkey@0"
I20260807 18:27:36.839414 27675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:36.861975 27675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:36.866117 27675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:36.866413 27675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:36.886978 27675 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:39099
I20260807 18:27:36.886997 27742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:39099 every 8 connection(s)
I20260807 18:27:36.888347 27675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:27:36.892145 27743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:36.893265 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 27675
I20260807 18:27:36.893570 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:27:36.895493 27655 external_mini_cluster.cc:1428] Setting key b4783ce84adb43a5ee7b7f7a230fb1ae
I20260807 18:27:36.903764 27743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76: Bootstrap starting.
I20260807 18:27:36.906704 27743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:36.907755 27743 log.cc:824] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:36.910526 27743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76: No bootstrap required, opened a new log
I20260807 18:27:36.918226 27743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c236e2df27a401aa2e12764dd1cce76" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 39099 } }
I20260807 18:27:36.918682 27743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:36.918815 27743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c236e2df27a401aa2e12764dd1cce76, State: Initialized, Role: FOLLOWER
I20260807 18:27:36.919518 27743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c236e2df27a401aa2e12764dd1cce76" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 39099 } }
I20260807 18:27:36.919768 27743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:27:36.919911 27743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:27:36.920172 27743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:36.922448 27743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c236e2df27a401aa2e12764dd1cce76" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 39099 } }
I20260807 18:27:36.922981 27743 leader_election.cc:302] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c236e2df27a401aa2e12764dd1cce76; no voters:
Aug 07 18:27:36 dist-test-slave-1xrh krb5kdc[27662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:36.923717 27743 leader_election.cc:288] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:27:36.923862 27748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:36.925400 27748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [term 1 LEADER]: Becoming Leader. State: Replica: 5c236e2df27a401aa2e12764dd1cce76, State: Running, Role: LEADER
I20260807 18:27:36.926137 27748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c236e2df27a401aa2e12764dd1cce76" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 39099 } }
I20260807 18:27:36.926316 27743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:27:36.930785 27750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c236e2df27a401aa2e12764dd1cce76. Latest consensus state: current_term: 1 leader_uuid: "5c236e2df27a401aa2e12764dd1cce76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c236e2df27a401aa2e12764dd1cce76" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 39099 } } }
I20260807 18:27:36.931203 27750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [sys.catalog]: This master's current role is: LEADER
I20260807 18:27:36.931761 27749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c236e2df27a401aa2e12764dd1cce76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c236e2df27a401aa2e12764dd1cce76" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 39099 } } }
I20260807 18:27:36.932080 27749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76 [sys.catalog]: This master's current role is: LEADER
I20260807 18:27:36.941283 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:36.899439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.27.1.254:39099)
0807 18:27:36.900319 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:36.900337 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:36.901480 (+ 1143us) server_negotiation.cc:407] Connection header received
0807 18:27:36.910383 (+ 8903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:36.910401 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:36.910789 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:36.911184 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:36.914146 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:36.915981 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:36.917801 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:36.918544 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:36.918740 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:36.925591 (+ 6851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:36.925603 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:36.925624 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:36.925674 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:36.931268 (+ 5594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:36.932535 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:36.932547 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:36.932559 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:36.932846 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:36.933569 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:36.933578 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:36.933583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:36.934626 (+ 1043us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:36.938444 (+ 3818us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:36.938852 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 18:27:36.939439 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":426,"threads_started":1}
I20260807 18:27:36.941342 27754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:27:36.948184 27754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:27:36.957989 27754 catalog_manager.cc:1347] Generated new cluster ID: e50b77c313614bfa9b12b921e0cb76b0
I20260807 18:27:36.958115 27754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:27:36.977797 27754 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:27:36.979097 27754 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:27:36.988679 27754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c236e2df27a401aa2e12764dd1cce76: Generated new TSK 0
I20260807 18:27:36.989603 27754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:27:37.044324 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:39099
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:37.400072 27769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:37.400382 27769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:37.400484 27769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:37.407943 27769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:37.408061 27769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:37.408236 27769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:27:37.418051 27769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:39099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:37.419405 27769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:37.420753 27769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:37.434914 27774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:37.434949 27778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:37.435374 27776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:37.435761 27769 server_base.cc:1034] running on GCE node
I20260807 18:27:37.436395 27769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:37.437469 27769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:37.438716 27769 hybrid_clock.cc:648] HybridClock initialized: now 1786127257438658 us; error 59 us; skew 500 ppm
I20260807 18:27:37.439079 27769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:37 dist-test-slave-1xrh krb5kdc[27662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127257, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:37.442516 27769 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:27:37.444432 27769 webserver.cc:460] Webserver started at http://127.27.1.193:38547/ using document root <none> and password file <none>
I20260807 18:27:37.445152 27769 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:37.445284 27769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:37.445619 27769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:37.448760 27769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "81369d2e635241e0a4019495f2ceaef0"
format_stamp: "Formatted at 2026-08-07 18:27:37 on dist-test-slave-1xrh"
server_key: "069aa56c948cd709d177e24f70fcbe46"
server_key_iv: "b5f5e6ed298d6984f4a634907d2e2dc2"
server_key_version: "encryptionkey@0"
I20260807 18:27:37.449719 27769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "81369d2e635241e0a4019495f2ceaef0"
format_stamp: "Formatted at 2026-08-07 18:27:37 on dist-test-slave-1xrh"
server_key: "069aa56c948cd709d177e24f70fcbe46"
server_key_iv: "b5f5e6ed298d6984f4a634907d2e2dc2"
server_key_version: "encryptionkey@0"
I20260807 18:27:37.454934 27769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260807 18:27:37.458336 27785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:37.459661 27769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:27:37.459837 27769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "81369d2e635241e0a4019495f2ceaef0"
format_stamp: "Formatted at 2026-08-07 18:27:37 on dist-test-slave-1xrh"
server_key: "069aa56c948cd709d177e24f70fcbe46"
server_key_iv: "b5f5e6ed298d6984f4a634907d2e2dc2"
server_key_version: "encryptionkey@0"
I20260807 18:27:37.460026 27769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:37.485282 27769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:37.489779 27769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:37.490074 27769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:37.491220 27769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:27:37.492621 27769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:37.492731 27769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:37.492885 27769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:37.492959 27769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:37.530653 27769 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:43593
I20260807 18:27:37.530742 27903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:43593 every 8 connection(s)
I20260807 18:27:37.531965 27769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:27:37.538867 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 27769
I20260807 18:27:37.539105 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:27:37.539824 27655 external_mini_cluster.cc:1428] Setting key 2cb08f46bea6fd23fb5dc8655ad6946c
Aug 07 18:27:37 dist-test-slave-1xrh krb5kdc[27662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127257, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:37.554100 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:37.534766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:44943 (local address 127.27.1.254:39099)
0807 18:27:37.535340 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:37.535345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:37.536214 (+ 869us) server_negotiation.cc:407] Connection header received
0807 18:27:37.538271 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:37.538277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:37.538383 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:37.538550 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:37.540422 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:37.541506 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:37.543761 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:37.544398 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:37.544556 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:37.548227 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:37.548232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:37.548236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:37.548269 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:37.550989 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:37.551775 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:37.551780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:37.551782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:37.551902 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:37.552335 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:37.552340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:37.552343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:37.552604 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:37.552760 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:37.553473 (+ 713us) server_negotiation.cc:299] Negotiation successful
0807 18:27:37.553687 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":255,"threads_started":1}
I20260807 18:27:37.555105 27907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:37.535303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:39099 (local address 127.27.1.193:44943)
0807 18:27:37.536036 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:37.536088 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:37.537986 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:37.538870 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:37.538879 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:37.539581 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:37.540235 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:37.540254 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:37.541710 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:37.541715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:37.543602 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:37.543614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:37.544645 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:37.544654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:37.544918 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:37.545628 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:37.545650 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:37.547998 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:37.551156 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:37.551165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:37.551182 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:37.551642 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:37.552012 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:37.552017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:37.552020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:37.552209 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:37.552741 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:37.552749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:37.553082 (+ 333us) client_negotiation.cc:769] Sending connection context
0807 18:27:37.553354 (+ 272us) client_negotiation.cc:240] Negotiation successful
0807 18:27:37.553956 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":285,"threads_started":1}
I20260807 18:27:37.556854 27904 heartbeater.cc:347] Connected to a master server at 127.27.1.254:39099
I20260807 18:27:37.557241 27904 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:37.558306 27904 heartbeater.cc:511] Master 127.27.1.254:39099 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
I20260807 18:27:37.561437 27707 ts_manager.cc:194] Registered new tserver with Master: 81369d2e635241e0a4019495f2ceaef0 (127.27.1.193:43593)
I20260807 18:27:37.563359 27707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:44943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:27:37.591351 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:39099
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:37.942171 27910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:37.942448 27910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:37.942613 27910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:37.950212 27910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:37.950320 27910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:37.950445 27910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:27:37.960276 27910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:39099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:37.961485 27910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:37.962816 27910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:37.979382 27916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:37.979329 27915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:37.979424 27918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:37.980106 27910 server_base.cc:1034] running on GCE node
I20260807 18:27:37.980744 27910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:37.981676 27910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:37.982879 27910 hybrid_clock.cc:648] HybridClock initialized: now 1786127257982819 us; error 50 us; skew 500 ppm
I20260807 18:27:37.983297 27910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:37 dist-test-slave-1xrh krb5kdc[27662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127257, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:37.986692 27910 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:27:37.988431 27910 webserver.cc:460] Webserver started at http://127.27.1.194:37773/ using document root <none> and password file <none>
I20260807 18:27:37.989403 27910 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:37.989540 27910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:37.989864 27910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:37.992908 27910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "5d12b58d2e594823928bffcdb1763674"
format_stamp: "Formatted at 2026-08-07 18:27:37 on dist-test-slave-1xrh"
server_key: "47b5f469fd341c19d1b253353f661320"
server_key_iv: "f4840e6647075a4f68364e47fc9092a7"
server_key_version: "encryptionkey@0"
I20260807 18:27:37.993855 27910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "5d12b58d2e594823928bffcdb1763674"
format_stamp: "Formatted at 2026-08-07 18:27:37 on dist-test-slave-1xrh"
server_key: "47b5f469fd341c19d1b253353f661320"
server_key_iv: "f4840e6647075a4f68364e47fc9092a7"
server_key_version: "encryptionkey@0"
I20260807 18:27:37.999375 27910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 18:27:38.002755 27925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:38.004115 27910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:27:38.004278 27910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "5d12b58d2e594823928bffcdb1763674"
format_stamp: "Formatted at 2026-08-07 18:27:37 on dist-test-slave-1xrh"
server_key: "47b5f469fd341c19d1b253353f661320"
server_key_iv: "f4840e6647075a4f68364e47fc9092a7"
server_key_version: "encryptionkey@0"
I20260807 18:27:38.004457 27910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:38.017042 27910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:38.021122 27910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:38.021418 27910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:38.022343 27910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:27:38.023857 27910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:38.023931 27910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:38.024112 27910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:38.024166 27910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:38.062721 27910 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:36989
I20260807 18:27:38.062758 28040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:36989 every 8 connection(s)
I20260807 18:27:38.064277 27910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:27:38.070466 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 27910
I20260807 18:27:38.070678 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:27:38.071179 27655 external_mini_cluster.cc:1428] Setting key 6d9fde43d71e3633fb98791f154c390a
Aug 07 18:27:38 dist-test-slave-1xrh krb5kdc[27662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127257, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:38.085909 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:38.066813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:51625 (local address 127.27.1.254:39099)
0807 18:27:38.067297 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:38.067301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:38.068258 (+ 957us) server_negotiation.cc:407] Connection header received
0807 18:27:38.070139 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:38.070144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:38.070240 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:38.070416 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:38.072733 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.073841 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.075454 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.076354 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.076507 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:38.079979 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:38.079984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:38.079987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:38.080019 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:38.082792 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.083623 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.083627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.083629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.083752 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.084199 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.084206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.084214 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.084464 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:38.084619 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:38.085307 (+ 688us) server_negotiation.cc:299] Negotiation successful
0807 18:27:38.085529 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":219,"threads_started":1}
I20260807 18:27:38.086962 28044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:38.067354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:39099 (local address 127.27.1.194:51625)
0807 18:27:38.068098 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:38.068135 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:38.069865 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:38.070916 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:38.070990 (+ 74us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:38.071771 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:38.072540 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:38.072559 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.074011 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.074015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:38.075301 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:38.075318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.076627 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.076633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:38.076835 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:38.077528 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:38.077550 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:38.079744 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:38.082935 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:38.082944 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:38.082961 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:38.083487 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:38.083878 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:38.083883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:38.083887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:38.084076 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:38.084593 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:38.084601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:38.084935 (+ 334us) client_negotiation.cc:769] Sending connection context
0807 18:27:38.085199 (+ 264us) client_negotiation.cc:240] Negotiation successful
0807 18:27:38.085721 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":332,"threads_started":1}
I20260807 18:27:38.088483 28041 heartbeater.cc:347] Connected to a master server at 127.27.1.254:39099
I20260807 18:27:38.088796 28041 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:38.089496 28041 heartbeater.cc:511] Master 127.27.1.254:39099 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
I20260807 18:27:38.090885 27707 ts_manager.cc:194] Registered new tserver with Master: 5d12b58d2e594823928bffcdb1763674 (127.27.1.194:36989)
I20260807 18:27:38.091871 27707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:51625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:27:38.114400 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:39099
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:38.460256 28047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:38.460592 28047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:38.460795 28047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:38.468174 28047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:38.468273 28047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:38.468487 28047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:27:38.478143 28047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:39099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:38.479447 28047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:38.480813 28047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:38.494122 28055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:38.494205 28052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:38.494227 28053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:38.495045 28047 server_base.cc:1034] running on GCE node
I20260807 18:27:38.495639 28047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:38.496541 28047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:38.497743 28047 hybrid_clock.cc:648] HybridClock initialized: now 1786127258497709 us; error 30 us; skew 500 ppm
I20260807 18:27:38.498039 28047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:38 dist-test-slave-1xrh krb5kdc[27662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127258, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:38.501262 28047 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:27:38.502691 28047 webserver.cc:460] Webserver started at http://127.27.1.195:40681/ using document root <none> and password file <none>
I20260807 18:27:38.503445 28047 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:38.503621 28047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:38.503985 28047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:38.507042 28047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "8fb30364ebee4697a55d3cfdb37fb9ae"
format_stamp: "Formatted at 2026-08-07 18:27:38 on dist-test-slave-1xrh"
server_key: "13b0f4fca1faccc53c1956d44134d4c4"
server_key_iv: "f4c8d48c64f0775a21642e765487109e"
server_key_version: "encryptionkey@0"
I20260807 18:27:38.507968 28047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "8fb30364ebee4697a55d3cfdb37fb9ae"
format_stamp: "Formatted at 2026-08-07 18:27:38 on dist-test-slave-1xrh"
server_key: "13b0f4fca1faccc53c1956d44134d4c4"
server_key_iv: "f4c8d48c64f0775a21642e765487109e"
server_key_version: "encryptionkey@0"
I20260807 18:27:38.513357 28047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:27:38.516957 28063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:38.518267 28047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 18:27:38.518394 28047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "8fb30364ebee4697a55d3cfdb37fb9ae"
format_stamp: "Formatted at 2026-08-07 18:27:38 on dist-test-slave-1xrh"
server_key: "13b0f4fca1faccc53c1956d44134d4c4"
server_key_iv: "f4c8d48c64f0775a21642e765487109e"
server_key_version: "encryptionkey@0"
I20260807 18:27:38.518566 28047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:38.559021 28047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:38.563966 28047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:38.564313 28047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:38.565392 28047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:27:38.566510 27904 heartbeater.cc:503] Master 127.27.1.254:39099 was elected leader, sending a full tablet report...
I20260807 18:27:38.566735 28047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:38.566813 28047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:38.566896 28047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:38.566932 28047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:38.604316 28047 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:35695
I20260807 18:27:38.604493 28177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:35695 every 8 connection(s)
I20260807 18:27:38.605693 28047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:27:38.613155 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 28047
I20260807 18:27:38.613359 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:27:38.613929 27655 external_mini_cluster.cc:1428] Setting key 399aded68bd0e6ef16337cfe6b1efeee
Aug 07 18:27:38 dist-test-slave-1xrh krb5kdc[27662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127258, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:38.626874 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:38.608348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:55607 (local address 127.27.1.254:39099)
0807 18:27:38.608830 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:38.608837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:38.609913 (+ 1076us) server_negotiation.cc:407] Connection header received
0807 18:27:38.611920 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:38.611924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:38.612027 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:38.612122 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:38.613655 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.614840 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.616658 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.617237 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.617385 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:38.620942 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:38.620947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:38.620951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:38.620992 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:38.623802 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.624521 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.624525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.624527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.624649 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.625098 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.625101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.625103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.625356 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:38.625529 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:38.626169 (+ 640us) server_negotiation.cc:299] Negotiation successful
0807 18:27:38.626384 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":231,"threads_started":1}
I20260807 18:27:38.627923 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:38.608942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:39099 (local address 127.27.1.195:55607)
0807 18:27:38.609675 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:38.609720 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:38.611673 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:38.612314 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:38.612322 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:38.612817 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:38.613466 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:38.613493 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.614995 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.615004 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:38.616529 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:38.616542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.617477 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.617484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:38.617787 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:38.618413 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:38.618435 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:38.620731 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:38.623938 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:38.623946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:38.623972 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:38.624371 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:38.624763 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:38.624768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:38.624773 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:38.624964 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:38.625494 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:38.625502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:38.625807 (+ 305us) client_negotiation.cc:769] Sending connection context
0807 18:27:38.626071 (+ 264us) client_negotiation.cc:240] Negotiation successful
0807 18:27:38.626624 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":330,"threads_started":1}
I20260807 18:27:38.629355 28178 heartbeater.cc:347] Connected to a master server at 127.27.1.254:39099
I20260807 18:27:38.629700 28178 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:38.630630 28178 heartbeater.cc:511] Master 127.27.1.254:39099 requested a full tablet report, sending...
I20260807 18:27:38.632150 27706 ts_manager.cc:194] Registered new tserver with Master: 8fb30364ebee4697a55d3cfdb37fb9ae (127.27.1.195:35695)
I20260807 18:27:38.633044 27706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:55607
I20260807 18:27:38.642362 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 18:27:38 dist-test-slave-1xrh krb5kdc[27662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127258, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 18:27:38 dist-test-slave-1xrh krb5kdc[27662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127258, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:27:38.693035 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:38.677005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.27.1.193:43593)
0807 18:27:38.677748 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:38.677755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:38.677781 (+ 26us) server_negotiation.cc:407] Connection header received
0807 18:27:38.677846 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:38.677852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:38.678096 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:38.678289 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:38.679885 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.680916 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.682180 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.682753 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.682888 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:38.687110 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:38.687163 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:38.687184 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:38.687232 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:38.690115 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.690788 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.690796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.690799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.690931 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.691395 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.691400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.691404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.691786 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:38.692098 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:38.692432 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 18:27:38.692706 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":345,"threads_started":1}
W20260807 18:27:38.694245 27813 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164
Aug 07 18:27:38 dist-test-slave-1xrh krb5kdc[27662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 18:27:38 dist-test-slave-1xrh krb5kdc[27662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:27:38.740242 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:38.727021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.27.1.193:43593)
0807 18:27:38.727285 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:38.727292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:38.727464 (+ 172us) server_negotiation.cc:407] Connection header received
0807 18:27:38.727635 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:38.727639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:38.727707 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:38.727859 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:38.729227 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.729898 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.731105 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.731584 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.731717 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:38.735557 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:38.735562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:38.735565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:38.735600 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:38.737935 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.738502 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.738506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.738508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.738625 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.738936 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.738939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.738941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.739241 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:38.739415 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:38.739669 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 18:27:38.739826 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260807 18:27:38.741878 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 27769
I20260807 18:27:38.754230 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:38.742933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.27.1.193:43593)
0807 18:27:38.743103 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:38.743107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:38.743380 (+ 273us) server_negotiation.cc:407] Connection header received
0807 18:27:38.743509 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:38.743512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:38.743567 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:38.743644 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:38.744828 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.745492 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.746749 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.747268 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.747362 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:38.749374 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:38.749378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:38.749382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:38.749409 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:38.752160 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.752706 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.752710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.752712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.752834 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.753130 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.753133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.753135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.753358 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:38.753555 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:38.753712 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 18:27:38.753863 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:27:38.754926 27899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:27:38.839373 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 27769
I20260807 18:27:38.859198 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 27910
Aug 07 18:27:38 dist-test-slave-1xrh krb5kdc[27662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:27:38.874221 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:38.860079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.27.1.194:36989)
0807 18:27:38.860616 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:38.860623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:38.860649 (+ 26us) server_negotiation.cc:407] Connection header received
0807 18:27:38.860713 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:38.860720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:38.860924 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:38.861058 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:38.862179 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.863205 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.864465 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.865021 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.865122 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:38.868986 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:38.868996 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:38.869009 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:38.869048 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:38.871734 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.872334 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.872341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.872344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.872495 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.872837 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.872840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.872842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.873168 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:38.873392 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:38.873631 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 18:27:38.873818 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":273,"threads_started":1}
I20260807 18:27:38.875643 28036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:27:38.961491 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 27910
I20260807 18:27:38.981187 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 28047
Aug 07 18:27:38 dist-test-slave-1xrh krb5kdc[27662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:27:38.997381 28195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:38.982234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.27.1.195:35695)
0807 18:27:38.982900 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:38.982907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:38.982939 (+ 32us) server_negotiation.cc:407] Connection header received
0807 18:27:38.983001 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:38.983008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:38.983329 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:38.983504 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:38.984831 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.985869 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.987334 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:38.987910 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:38.988011 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:38.992146 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:38.992157 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:38.992171 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:38.992218 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:38.995002 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.995645 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.995652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.995654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.995773 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:38.996088 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:38.996091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:38.996093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:38.996419 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:38.996651 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:38.996888 (+ 237us) server_negotiation.cc:299] Negotiation successful
0807 18:27:38.997101 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":330,"threads_started":1}
I20260807 18:27:38.998657 28173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:27:39.081753 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 28047
I20260807 18:27:39.100775 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 27675
I20260807 18:27:39.101964 27738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:27:39.207230 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 27675
2026-08-07T18:27:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4150 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:27:39 dist-test-slave-1xrh krb5kdc[28200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:27:39 dist-test-slave-1xrh krb5kdc[28200](info): set up 2 sockets
Aug 07 18:27:39 dist-test-slave-1xrh krb5kdc[28200](info): commencing operation
krb5kdc: starting...
W20260807 18:27:40.341351 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:27:40 dist-test-slave-1xrh krb5kdc[28200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:27:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:27:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:27:40.547619 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:40621
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:40.883948 28214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:40.884295 28214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:40.884456 28214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:40.891618 28214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:27:40.891755 28214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:40.891865 28214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:40.891978 28214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:27:40.892067 28214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:27:40.901744 28214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:40621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:40.903080 28214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:40.904428 28214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:40.916630 28219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:40.916596 28220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:40.916674 28222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:40.917579 28214 server_base.cc:1034] running on GCE node
I20260807 18:27:40.918360 28214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:40.919804 28214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:40.921020 28214 hybrid_clock.cc:648] HybridClock initialized: now 1786127260920973 us; error 31 us; skew 500 ppm
I20260807 18:27:40.921376 28214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:40 dist-test-slave-1xrh krb5kdc[28200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127260, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:40.924621 28214 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:27:40.926070 28214 webserver.cc:460] Webserver started at http://127.27.1.254:37647/ using document root <none> and password file <none>
I20260807 18:27:40.926784 28214 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:40.926884 28214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:40.927289 28214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:40.930300 28214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "cdef0355d3fa432cb705f4c9ca4bdce4"
format_stamp: "Formatted at 2026-08-07 18:27:40 on dist-test-slave-1xrh"
server_key: "d29db0c62535ab9f85a48db0bd4f6b3e"
server_key_iv: "6b6d6dae61eb8a4d11d0ac797f74a3a7"
server_key_version: "encryptionkey@0"
I20260807 18:27:40.931171 28214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "cdef0355d3fa432cb705f4c9ca4bdce4"
format_stamp: "Formatted at 2026-08-07 18:27:40 on dist-test-slave-1xrh"
server_key: "d29db0c62535ab9f85a48db0bd4f6b3e"
server_key_iv: "6b6d6dae61eb8a4d11d0ac797f74a3a7"
server_key_version: "encryptionkey@0"
I20260807 18:27:40.936386 28214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260807 18:27:40.939812 28229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:40.941097 28214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:27:40.941282 28214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "cdef0355d3fa432cb705f4c9ca4bdce4"
format_stamp: "Formatted at 2026-08-07 18:27:40 on dist-test-slave-1xrh"
server_key: "d29db0c62535ab9f85a48db0bd4f6b3e"
server_key_iv: "6b6d6dae61eb8a4d11d0ac797f74a3a7"
server_key_version: "encryptionkey@0"
I20260807 18:27:40.941465 28214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:40.958853 28214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:40.962739 28214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:40.963043 28214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:40.982164 28214 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:40621
I20260807 18:27:40.982203 28281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:40621 every 8 connection(s)
I20260807 18:27:40.983667 28214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:27:40.986811 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 28214
I20260807 18:27:40.987051 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:27:40.987949 27655 external_mini_cluster.cc:1428] Setting key f8b79aec0f1f81b5af8ea79a97654114
I20260807 18:27:40.987977 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:41.000859 28282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4: Bootstrap starting.
Aug 07 18:27:40 dist-test-slave-1xrh krb5kdc[28200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:41.004462 28282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:41.005707 28282 log.cc:824] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:41.008646 28282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4: No bootstrap required, opened a new log
I20260807 18:27:41.010511 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:40.990053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.27.1.254:40621)
0807 18:27:40.990819 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:40.990831 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:40.990875 (+ 44us) server_negotiation.cc:407] Connection header received
0807 18:27:40.992903 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:40.992931 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:40.993355 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:40.993773 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:40.995347 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:40.996650 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:40.997879 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:40.998581 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:40.998815 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:41.003081 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:41.003088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:41.003102 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:41.003196 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:41.006487 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:41.007199 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:41.007206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:41.007212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:41.007370 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:41.007740 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:41.007745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:41.007749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:41.008267 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:41.008571 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:41.009129 (+ 558us) server_negotiation.cc:299] Negotiation successful
0807 18:27:41.009550 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":348,"threads_started":1}
I20260807 18:27:41.016673 28282 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdef0355d3fa432cb705f4c9ca4bdce4" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 40621 } }
I20260807 18:27:41.017155 28282 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:41.017299 28282 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdef0355d3fa432cb705f4c9ca4bdce4, State: Initialized, Role: FOLLOWER
I20260807 18:27:41.017957 28282 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdef0355d3fa432cb705f4c9ca4bdce4" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 40621 } }
I20260807 18:27:41.018170 28282 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:27:41.018275 28282 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:27:41.018396 28282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:41.020569 28282 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdef0355d3fa432cb705f4c9ca4bdce4" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 40621 } }
I20260807 18:27:41.020998 28282 leader_election.cc:302] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdef0355d3fa432cb705f4c9ca4bdce4; no voters:
I20260807 18:27:41.021564 28282 leader_election.cc:288] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:27:41.021734 28287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:41.022554 28287 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [term 1 LEADER]: Becoming Leader. State: Replica: cdef0355d3fa432cb705f4c9ca4bdce4, State: Running, Role: LEADER
I20260807 18:27:41.023056 28287 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdef0355d3fa432cb705f4c9ca4bdce4" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 40621 } }
I20260807 18:27:41.023932 28282 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:27:41.026253 28289 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdef0355d3fa432cb705f4c9ca4bdce4. Latest consensus state: current_term: 1 leader_uuid: "cdef0355d3fa432cb705f4c9ca4bdce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdef0355d3fa432cb705f4c9ca4bdce4" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 40621 } } }
I20260807 18:27:41.026645 28289 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [sys.catalog]: This master's current role is: LEADER
I20260807 18:27:41.026327 28288 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdef0355d3fa432cb705f4c9ca4bdce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdef0355d3fa432cb705f4c9ca4bdce4" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 40621 } } }
I20260807 18:27:41.026804 28288 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4 [sys.catalog]: This master's current role is: LEADER
W20260807 18:27:41.035786 28303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 18:27:41.035902 28303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 18:27:41.036060 28293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:27:41.037914 28293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:27:41.049607 28293 catalog_manager.cc:1347] Generated new cluster ID: fb2da80c3b43403780ac0ca67d6fe7e9
I20260807 18:27:41.049731 28293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:27:41.080680 28293 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:27:41.082597 28293 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:27:41.091593 28293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdef0355d3fa432cb705f4c9ca4bdce4: Generated new TSK 0
I20260807 18:27:41.092602 28293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:27:41.149261 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:40621
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:41.484900 28308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:41.485225 28308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:41.485406 28308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:41.492588 28308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:41.492738 28308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:41.492929 28308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:27:41.502504 28308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:40621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:41.503847 28308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:41.505256 28308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:41.519476 28313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:41.519502 28314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:41.519410 28316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:41.520249 28308 server_base.cc:1034] running on GCE node
I20260807 18:27:41.520836 28308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:41.521802 28308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:41.523036 28308 hybrid_clock.cc:648] HybridClock initialized: now 1786127261522990 us; error 37 us; skew 500 ppm
I20260807 18:27:41.523404 28308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:41 dist-test-slave-1xrh krb5kdc[28200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127261, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:41.526860 28308 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:27:41.528625 28308 webserver.cc:460] Webserver started at http://127.27.1.193:33651/ using document root <none> and password file <none>
I20260807 18:27:41.529409 28308 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:41.529551 28308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:41.529892 28308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:41.533020 28308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "f535e143b41e4725b27fccf6d7c9d4dc"
format_stamp: "Formatted at 2026-08-07 18:27:41 on dist-test-slave-1xrh"
server_key: "6a1538539d0553f3bca4649723837ea7"
server_key_iv: "02e81125dff7b781b6c303fd8cb7a3c5"
server_key_version: "encryptionkey@0"
I20260807 18:27:41.533834 28308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "f535e143b41e4725b27fccf6d7c9d4dc"
format_stamp: "Formatted at 2026-08-07 18:27:41 on dist-test-slave-1xrh"
server_key: "6a1538539d0553f3bca4649723837ea7"
server_key_iv: "02e81125dff7b781b6c303fd8cb7a3c5"
server_key_version: "encryptionkey@0"
I20260807 18:27:41.539095 28308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 18:27:41.542557 28323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:41.543987 28308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:27:41.544166 28308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "f535e143b41e4725b27fccf6d7c9d4dc"
format_stamp: "Formatted at 2026-08-07 18:27:41 on dist-test-slave-1xrh"
server_key: "6a1538539d0553f3bca4649723837ea7"
server_key_iv: "02e81125dff7b781b6c303fd8cb7a3c5"
server_key_version: "encryptionkey@0"
I20260807 18:27:41.544364 28308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:41.557000 28308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:41.561035 28308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:41.561334 28308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:41.562265 28308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:27:41.564117 28308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:41.564222 28308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:41.564370 28308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:41.564445 28308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:41.601078 28308 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:40231
I20260807 18:27:41.601169 28445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:40231 every 8 connection(s)
I20260807 18:27:41.602378 28308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:27:41.610054 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 28308
I20260807 18:27:41.610323 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:27:41.611029 27655 external_mini_cluster.cc:1428] Setting key 403f1279b72f79d9968e4ebd09a9548d
Aug 07 18:27:41 dist-test-slave-1xrh krb5kdc[28200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127261, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:41.624115 28448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:41.605253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:48995 (local address 127.27.1.254:40621)
0807 18:27:41.605754 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:41.605759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:41.606522 (+ 763us) server_negotiation.cc:407] Connection header received
0807 18:27:41.608420 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:41.608426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:41.608519 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:41.608644 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:41.610223 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:41.611026 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:41.612788 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:41.613238 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:41.613367 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:41.617173 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:41.617182 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:41.617189 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:41.617232 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:41.620581 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:41.621290 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:41.621300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:41.621307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:41.621482 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:41.621910 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:41.621918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:41.621926 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:41.622347 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:41.622519 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:41.623203 (+ 684us) server_negotiation.cc:299] Negotiation successful
0807 18:27:41.623565 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":275,"threads_started":1}
I20260807 18:27:41.624818 28449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:41.605638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:40621 (local address 127.27.1.193:48995)
0807 18:27:41.606343 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:41.606394 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:41.608146 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:41.608873 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:41.608881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:41.609414 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:41.609953 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:41.609970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:41.611236 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:41.611241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:41.612632 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:41.612650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:41.613499 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:41.613505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:41.613729 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:41.614503 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:41.614528 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:41.616937 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:41.620724 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:41.620730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:41.620742 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:41.621146 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:41.621595 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:41.621599 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:41.621601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:41.621780 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:41.622481 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:41.622486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:41.622779 (+ 293us) client_negotiation.cc:769] Sending connection context
0807 18:27:41.622996 (+ 217us) client_negotiation.cc:240] Negotiation successful
0807 18:27:41.623482 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":291,"threads_started":1}
I20260807 18:27:41.626457 28446 heartbeater.cc:347] Connected to a master server at 127.27.1.254:40621
I20260807 18:27:41.626775 28446 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:41.627548 28446 heartbeater.cc:511] Master 127.27.1.254:40621 requested a full tablet report, sending...
I20260807 18:27:41.629798 28246 ts_manager.cc:194] Registered new tserver with Master: f535e143b41e4725b27fccf6d7c9d4dc (127.27.1.193:40231)
I20260807 18:27:41.631309 28246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:48995
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:27:41.660530 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:40621
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:41.984774 28452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:41.985097 28452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:41.985272 28452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:41.992667 28452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:41.992823 28452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:41.993021 28452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:27:42.002923 28452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:40621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:42.004345 28452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:42.005715 28452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:42.019387 28457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:42.019352 28458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:42.019352 28460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:42.020418 28452 server_base.cc:1034] running on GCE node
I20260807 18:27:42.020972 28452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:42.021895 28452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:42.023165 28452 hybrid_clock.cc:648] HybridClock initialized: now 1786127262023051 us; error 53 us; skew 500 ppm
I20260807 18:27:42.023536 28452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:42 dist-test-slave-1xrh krb5kdc[28200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127262, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:42.033030 28452 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:27:42.034524 28452 webserver.cc:460] Webserver started at http://127.27.1.194:38475/ using document root <none> and password file <none>
I20260807 18:27:42.035269 28452 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:42.035383 28452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:42.035641 28452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:42.038532 28452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "517e9795228541b58fc43b37dea6a4c7"
format_stamp: "Formatted at 2026-08-07 18:27:42 on dist-test-slave-1xrh"
server_key: "ddcabda4c0eaf71ae369e8bc42294294"
server_key_iv: "5d038ed4c0abe8a6a24e8e9cae0d5716"
server_key_version: "encryptionkey@0"
I20260807 18:27:42.039384 28452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "517e9795228541b58fc43b37dea6a4c7"
format_stamp: "Formatted at 2026-08-07 18:27:42 on dist-test-slave-1xrh"
server_key: "ddcabda4c0eaf71ae369e8bc42294294"
server_key_iv: "5d038ed4c0abe8a6a24e8e9cae0d5716"
server_key_version: "encryptionkey@0"
I20260807 18:27:42.044663 28452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:27:42.048206 28467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:42.049685 28452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:27:42.049860 28452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "517e9795228541b58fc43b37dea6a4c7"
format_stamp: "Formatted at 2026-08-07 18:27:42 on dist-test-slave-1xrh"
server_key: "ddcabda4c0eaf71ae369e8bc42294294"
server_key_iv: "5d038ed4c0abe8a6a24e8e9cae0d5716"
server_key_version: "encryptionkey@0"
I20260807 18:27:42.050066 28452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:42.080873 28452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:42.085234 28452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:42.085546 28452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:42.086630 28452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:27:42.088013 28452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:42.088090 28452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:42.088241 28452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:42.088312 28452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:42.122488 28452 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:36381
I20260807 18:27:42.122526 28583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:36381 every 8 connection(s)
I20260807 18:27:42.123957 28452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:27:42.129675 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 28452
I20260807 18:27:42.129925 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:27:42.130507 27655 external_mini_cluster.cc:1428] Setting key f7e0978eeac0dd30c943c296680368be
Aug 07 18:27:42 dist-test-slave-1xrh krb5kdc[28200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127262, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:42.148929 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.126820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:48987 (local address 127.27.1.254:40621)
0807 18:27:42.127333 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.127339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.128206 (+ 867us) server_negotiation.cc:407] Connection header received
0807 18:27:42.130677 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.130682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.130751 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.130848 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:42.132704 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.133486 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.136108 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.136530 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.136624 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.140787 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:42.140796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:42.140805 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:42.140852 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:42.144770 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.145933 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.145941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.145947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.146174 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.146655 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.146662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.146669 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.147063 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:42.147265 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.148021 (+ 756us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.148304 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":310,"threads_started":1}
I20260807 18:27:42.149596 28587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.127334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:40621 (local address 127.27.1.194:48987)
0807 18:27:42.128034 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:42.128080 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:42.130346 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:42.131168 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:42.131180 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:42.131804 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:42.132542 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.132562 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.133655 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.133663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.135961 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.135981 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.136762 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.136771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.137098 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.137834 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:42.137867 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:42.140541 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:42.144921 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:42.144928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:42.144941 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:42.145732 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:42.146311 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:42.146315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:42.146317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:42.146527 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:42.147220 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:42.147227 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:42.147593 (+ 366us) client_negotiation.cc:769] Sending connection context
0807 18:27:42.147900 (+ 307us) client_negotiation.cc:240] Negotiation successful
0807 18:27:42.148286 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":290,"threads_started":1}
I20260807 18:27:42.151197 28584 heartbeater.cc:347] Connected to a master server at 127.27.1.254:40621
I20260807 18:27:42.151515 28584 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:42.152251 28584 heartbeater.cc:511] Master 127.27.1.254:40621 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
I20260807 18:27:42.153760 28246 ts_manager.cc:194] Registered new tserver with Master: 517e9795228541b58fc43b37dea6a4c7 (127.27.1.194:36381)
I20260807 18:27:42.154733 28246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:48987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:27:42.179203 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:40621
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:42.511863 28590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:42.512142 28590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:42.512239 28590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:42.519194 28590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:42.519294 28590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:42.519404 28590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:27:42.529024 28590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:40621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:42.530331 28590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:42.531728 28590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:42.544864 28596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:42.544945 28598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:42.544919 28595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:42.545940 28590 server_base.cc:1034] running on GCE node
I20260807 18:27:42.546566 28590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:42.547698 28590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:42.548955 28590 hybrid_clock.cc:648] HybridClock initialized: now 1786127262548916 us; error 85 us; skew 500 ppm
I20260807 18:27:42.549288 28590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:42 dist-test-slave-1xrh krb5kdc[28200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127262, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:42.552488 28590 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:27:42.554109 28590 webserver.cc:460] Webserver started at http://127.27.1.195:39041/ using document root <none> and password file <none>
I20260807 18:27:42.554893 28590 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:42.555044 28590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:42.555387 28590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:42.558737 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "7ac496eb18224aa594f9f9bc74fac5d6"
format_stamp: "Formatted at 2026-08-07 18:27:42 on dist-test-slave-1xrh"
server_key: "b6eeae917b7d2d1c9449a8d987288b82"
server_key_iv: "6e7c9d9fa5c0da7071e35575d960169c"
server_key_version: "encryptionkey@0"
I20260807 18:27:42.559589 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "7ac496eb18224aa594f9f9bc74fac5d6"
format_stamp: "Formatted at 2026-08-07 18:27:42 on dist-test-slave-1xrh"
server_key: "b6eeae917b7d2d1c9449a8d987288b82"
server_key_iv: "6e7c9d9fa5c0da7071e35575d960169c"
server_key_version: "encryptionkey@0"
I20260807 18:27:42.564837 28590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:27:42.568377 28606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:42.569722 28590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 18:27:42.569934 28590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "7ac496eb18224aa594f9f9bc74fac5d6"
format_stamp: "Formatted at 2026-08-07 18:27:42 on dist-test-slave-1xrh"
server_key: "b6eeae917b7d2d1c9449a8d987288b82"
server_key_iv: "6e7c9d9fa5c0da7071e35575d960169c"
server_key_version: "encryptionkey@0"
I20260807 18:27:42.570140 28590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:42.588778 28590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:42.593118 28590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:42.593427 28590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:42.594460 28590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:27:42.595866 28590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:42.595985 28590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:42.596141 28590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:42.596223 28590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:42.632637 28590 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:45173
I20260807 18:27:42.632683 28728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:45173 every 8 connection(s)
I20260807 18:27:42.633919 28590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:27:42.634301 28446 heartbeater.cc:503] Master 127.27.1.254:40621 was elected leader, sending a full tablet report...
I20260807 18:27:42.638166 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 28590
I20260807 18:27:42.638423 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:27:42.638963 27655 external_mini_cluster.cc:1428] Setting key 9cc484bb51570736be6382f3ad02a1a8
Aug 07 18:27:42 dist-test-slave-1xrh krb5kdc[28200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127262, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:42.656983 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.637266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:33919 (local address 127.27.1.254:40621)
0807 18:27:42.637447 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.637452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.639067 (+ 1615us) server_negotiation.cc:407] Connection header received
0807 18:27:42.641921 (+ 2854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.641925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.642062 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.642159 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:42.643996 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.644774 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.646579 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.646970 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.647067 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.650849 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:42.650854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:42.650857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:42.650888 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:42.653919 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.654626 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.654631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.654635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.654808 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.655470 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.655474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.655476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.655737 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:42.655871 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.656590 (+ 719us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.656750 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 18:27:42.658097 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.637836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:40621 (local address 127.27.1.195:33919)
0807 18:27:42.638897 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:42.638942 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:42.641601 (+ 2659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:42.642438 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:42.642448 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:42.643171 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:42.643824 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.643857 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.644947 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.644952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.646446 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.646459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.647262 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.647268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.647559 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.648258 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:42.648285 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:42.650629 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:42.654050 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:42.654059 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:42.654075 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:42.654489 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:42.655047 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:42.655053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:42.655057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:42.655313 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:42.655876 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:42.655885 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:42.656211 (+ 326us) client_negotiation.cc:769] Sending connection context
0807 18:27:42.656496 (+ 285us) client_negotiation.cc:240] Negotiation successful
0807 18:27:42.656899 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":452,"threads_started":1}
I20260807 18:27:42.659519 28729 heartbeater.cc:347] Connected to a master server at 127.27.1.254:40621
I20260807 18:27:42.659844 28729 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:42.660674 28729 heartbeater.cc:511] Master 127.27.1.254:40621 requested a full tablet report, sending...
I20260807 18:27:42.662144 28246 ts_manager.cc:194] Registered new tserver with Master: 7ac496eb18224aa594f9f9bc74fac5d6 (127.27.1.195:45173)
I20260807 18:27:42.663038 28246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:33919
I20260807 18:27:42.667063 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:27:42.688635 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.674066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.27.1.254:40621)
0807 18:27:42.674288 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.674298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.674879 (+ 581us) server_negotiation.cc:407] Connection header received
0807 18:27:42.675101 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.675105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.675259 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.675357 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:42.677277 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.678583 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.680263 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.680708 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.680821 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.683305 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:42.683310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:42.683313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:42.683350 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:42.685910 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.686530 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.686535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.686538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.686685 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.687036 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.687039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.687041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.687461 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:42.687674 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.688057 (+ 383us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.688226 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:27:42.695642 28246 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:27:42.703426 28246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:27:42.742074 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.718738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.27.1.194:36381)
0807 18:27:42.725893 (+ 7155us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.725908 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.732276 (+ 6368us) server_negotiation.cc:407] Connection header received
0807 18:27:42.732672 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.732684 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.732992 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.733210 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:42.734049 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.735654 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.739277 (+ 3623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.740817 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.740958 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.741154 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.741665 (+ 511us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.741796 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6887,"thread_start_us":6715,"threads_started":1}
I20260807 18:27:42.742323 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.717122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:36381 (local address 127.0.0.1:37812)
0807 18:27:42.732103 (+ 14981us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:42.732128 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:42.732419 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:42.733285 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:42.733289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:42.733319 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:42.733818 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.733831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.735869 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.735874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.739070 (+ 3196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.739086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.741164 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.741175 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.741399 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.741459 (+ 60us) client_negotiation.cc:769] Sending connection context
0807 18:27:42.741614 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 18:27:42.741835 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14784,"mutex_wait_us":180,"thread_start_us":14419,"threads_started":1}
I20260807 18:27:42.743355 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.716358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40231 (local address 127.0.0.1:56998)
0807 18:27:42.730794 (+ 14436us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:42.730850 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:42.731076 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:42.734597 (+ 3521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:42.734601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:42.734629 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:42.735179 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.735193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.737021 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.737027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.739501 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.739813 (+ 312us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.742432 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.742439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.742650 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.742682 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 18:27:42.742795 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 18:27:42.742937 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14175,"mutex_wait_us":51,"thread_start_us":15551,"threads_started":1}
I20260807 18:27:42.743407 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.717070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.27.1.193:40231)
0807 18:27:42.734131 (+ 17061us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.734138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.734160 (+ 22us) server_negotiation.cc:407] Connection header received
0807 18:27:42.734237 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.734242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.734405 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.734608 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:42.735412 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.736854 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.740096 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.741490 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.741649 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.741981 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.742860 (+ 879us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.743026 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16876,"thread_start_us":16675,"threads_started":1}
I20260807 18:27:42.743731 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.718064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.27.1.195:45173)
0807 18:27:42.726675 (+ 8611us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.726688 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.731875 (+ 5187us) server_negotiation.cc:407] Connection header received
0807 18:27:42.732393 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.732401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.732690 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.732863 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:42.733844 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.735852 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.739294 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.740817 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.740956 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.741195 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.742678 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.742887 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8367,"thread_start_us":8094,"threads_started":1}
I20260807 18:27:42.744619 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.716855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:45173 (local address 127.0.0.1:41612)
0807 18:27:42.731653 (+ 14798us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:42.731671 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:42.732169 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:42.733096 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:42.733106 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:42.733135 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:42.733587 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.733599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.736014 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.736019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.739071 (+ 3052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.739086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.742161 (+ 3075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.742168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.742426 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.742467 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 18:27:42.743990 (+ 1523us) client_negotiation.cc:240] Negotiation successful
0807 18:27:42.744135 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14644,"mutex_wait_us":264,"thread_start_us":13517,"threads_started":1}
I20260807 18:27:42.746814 28517 tablet_service.cc:1467] Processing CreateTablet for tablet e33d3e11b08c487bb2470cc6f341e653 (DEFAULT_TABLE table=test-table [id=3e239072398a47ab849bfae5904be54b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:42.748446 28517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33d3e11b08c487bb2470cc6f341e653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:42.748193 28661 tablet_service.cc:1467] Processing CreateTablet for tablet e33d3e11b08c487bb2470cc6f341e653 (DEFAULT_TABLE table=test-table [id=3e239072398a47ab849bfae5904be54b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:42.749853 28661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33d3e11b08c487bb2470cc6f341e653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:42.751273 28380 tablet_service.cc:1467] Processing CreateTablet for tablet e33d3e11b08c487bb2470cc6f341e653 (DEFAULT_TABLE table=test-table [id=3e239072398a47ab849bfae5904be54b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:42.752867 28380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33d3e11b08c487bb2470cc6f341e653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:42.760533 28748 tablet_bootstrap.cc:492] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7: Bootstrap starting.
I20260807 18:27:42.764386 28748 tablet_bootstrap.cc:654] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:42.765457 28748 log.cc:824] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:42.766681 28749 tablet_bootstrap.cc:492] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6: Bootstrap starting.
I20260807 18:27:42.769753 28751 tablet_bootstrap.cc:492] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc: Bootstrap starting.
I20260807 18:27:42.770399 28748 tablet_bootstrap.cc:492] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7: No bootstrap required, opened a new log
I20260807 18:27:42.770689 28748 ts_tablet_manager.cc:1397] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260807 18:27:42.771452 28749 tablet_bootstrap.cc:654] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:42.773149 28749 log.cc:824] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:42.773540 28751 tablet_bootstrap.cc:654] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:42.774950 28751 log.cc:824] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:42.776696 28749 tablet_bootstrap.cc:492] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6: No bootstrap required, opened a new log
I20260807 18:27:42.776994 28749 ts_tablet_manager.cc:1397] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260807 18:27:42.777870 28751 tablet_bootstrap.cc:492] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc: No bootstrap required, opened a new log
I20260807 18:27:42.778153 28751 ts_tablet_manager.cc:1397] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260807 18:27:42.781643 28748 raft_consensus.cc:348] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40231 } } peers { permanent_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45173 } } peers { permanent_uuid: "517e9795228541b58fc43b37dea6a4c7" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 36381 } }
I20260807 18:27:42.782143 28748 raft_consensus.cc:374] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:42.782245 28748 raft_consensus.cc:729] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 517e9795228541b58fc43b37dea6a4c7, State: Initialized, Role: FOLLOWER
I20260807 18:27:42.783018 28748 consensus_queue.cc:260] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40231 } } peers { permanent_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45173 } } peers { permanent_uuid: "517e9795228541b58fc43b37dea6a4c7" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 36381 } }
I20260807 18:27:42.784358 28584 heartbeater.cc:503] Master 127.27.1.254:40621 was elected leader, sending a full tablet report...
I20260807 18:27:42.785115 28748 ts_tablet_manager.cc:1428] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260807 18:27:42.787917 28751 raft_consensus.cc:348] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40231 } } peers { permanent_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45173 } } peers { permanent_uuid: "517e9795228541b58fc43b37dea6a4c7" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 36381 } }
I20260807 18:27:42.788358 28751 raft_consensus.cc:374] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:42.788492 28751 raft_consensus.cc:729] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f535e143b41e4725b27fccf6d7c9d4dc, State: Initialized, Role: FOLLOWER
I20260807 18:27:42.788292 28749 raft_consensus.cc:348] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40231 } } peers { permanent_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45173 } } peers { permanent_uuid: "517e9795228541b58fc43b37dea6a4c7" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 36381 } }
I20260807 18:27:42.788759 28749 raft_consensus.cc:374] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:42.788874 28749 raft_consensus.cc:729] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac496eb18224aa594f9f9bc74fac5d6, State: Initialized, Role: FOLLOWER
I20260807 18:27:42.789386 28751 consensus_queue.cc:260] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40231 } } peers { permanent_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45173 } } peers { permanent_uuid: "517e9795228541b58fc43b37dea6a4c7" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 36381 } }
I20260807 18:27:42.789568 28749 consensus_queue.cc:260] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40231 } } peers { permanent_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45173 } } peers { permanent_uuid: "517e9795228541b58fc43b37dea6a4c7" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 36381 } }
I20260807 18:27:42.791610 28751 ts_tablet_manager.cc:1428] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260807 18:27:42.792780 28729 heartbeater.cc:503] Master 127.27.1.254:40621 was elected leader, sending a full tablet report...
I20260807 18:27:42.793383 28749 ts_tablet_manager.cc:1428] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260807 18:27:42.801082 28754 raft_consensus.cc:482] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:42.801397 28754 raft_consensus.cc:504] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40231 } } peers { permanent_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45173 } } peers { permanent_uuid: "517e9795228541b58fc43b37dea6a4c7" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 36381 } }
I20260807 18:27:42.803167 28754 leader_election.cc:288] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f535e143b41e4725b27fccf6d7c9d4dc (127.27.1.193:40231), 7ac496eb18224aa594f9f9bc74fac5d6 (127.27.1.195:45173)
I20260807 18:27:42.810045 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.803538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:51975 (local address 127.27.1.193:40231)
0807 18:27:42.803768 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.803776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.804810 (+ 1034us) server_negotiation.cc:407] Connection header received
0807 18:27:42.805029 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.805034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.805101 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.805198 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:42.805709 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.806455 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.808326 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.809114 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.809302 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.809377 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.809646 (+ 269us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.809773 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 18:27:42.810138 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.803583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40231 (local address 127.27.1.194:51975)
0807 18:27:42.804694 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:42.804716 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:42.804889 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:42.805214 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:42.805224 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:42.805251 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:42.805569 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.805577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.806588 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.806595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.808171 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.808184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.809330 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.809337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.809486 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.809509 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 18:27:42.809618 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 18:27:42.809746 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":705,"threads_started":1}
I20260807 18:27:42.811070 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.803901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:41011 (local address 127.27.1.195:45173)
0807 18:27:42.804155 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.804165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.805077 (+ 912us) server_negotiation.cc:407] Connection header received
0807 18:27:42.805284 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.805289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.805368 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.805471 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:42.806095 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.806981 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.808926 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.809765 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.809994 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.810070 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.810661 (+ 591us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.810801 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 18:27:42.811115 28758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.803650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:45173 (local address 127.27.1.194:41011)
0807 18:27:42.804966 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:42.804982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:42.805144 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:42.805495 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:42.805499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:42.805536 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:42.805932 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.805941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.807225 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.807229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.808748 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:42.808762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.810010 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.810017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:42.810466 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.810506 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 18:27:42.810620 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 18:27:42.810751 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"thread_start_us":873,"threads_started":1}
I20260807 18:27:42.811499 28401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33d3e11b08c487bb2470cc6f341e653" candidate_uuid: "517e9795228541b58fc43b37dea6a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" is_pre_election: true
I20260807 18:27:42.812135 28401 raft_consensus.cc:2454] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 517e9795228541b58fc43b37dea6a4c7 in term 0.
I20260807 18:27:42.812254 28682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33d3e11b08c487bb2470cc6f341e653" candidate_uuid: "517e9795228541b58fc43b37dea6a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" is_pre_election: true
I20260807 18:27:42.812826 28682 raft_consensus.cc:2454] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 517e9795228541b58fc43b37dea6a4c7 in term 0.
I20260807 18:27:42.812916 28471 leader_election.cc:302] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 517e9795228541b58fc43b37dea6a4c7, f535e143b41e4725b27fccf6d7c9d4dc; no voters:
I20260807 18:27:42.813390 28754 raft_consensus.cc:2781] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:27:42.813596 28754 raft_consensus.cc:482] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:27:42.813735 28754 raft_consensus.cc:3037] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:42.816470 28754 raft_consensus.cc:504] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40231 } } peers { permanent_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45173 } } peers { permanent_uuid: "517e9795228541b58fc43b37dea6a4c7" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 36381 } }
I20260807 18:27:42.817183 28754 leader_election.cc:288] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers f535e143b41e4725b27fccf6d7c9d4dc (127.27.1.193:40231), 7ac496eb18224aa594f9f9bc74fac5d6 (127.27.1.195:45173)
I20260807 18:27:42.817705 28682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33d3e11b08c487bb2470cc6f341e653" candidate_uuid: "517e9795228541b58fc43b37dea6a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac496eb18224aa594f9f9bc74fac5d6"
I20260807 18:27:42.817718 28401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33d3e11b08c487bb2470cc6f341e653" candidate_uuid: "517e9795228541b58fc43b37dea6a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f535e143b41e4725b27fccf6d7c9d4dc"
I20260807 18:27:42.817898 28682 raft_consensus.cc:3037] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:42.817939 28401 raft_consensus.cc:3037] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:42.820250 28682 raft_consensus.cc:2454] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 517e9795228541b58fc43b37dea6a4c7 in term 1.
I20260807 18:27:42.820525 28401 raft_consensus.cc:2454] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 517e9795228541b58fc43b37dea6a4c7 in term 1.
I20260807 18:27:42.820693 28468 leader_election.cc:302] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 517e9795228541b58fc43b37dea6a4c7, 7ac496eb18224aa594f9f9bc74fac5d6; no voters:
I20260807 18:27:42.821022 28754 raft_consensus.cc:2781] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:42.821527 28754 raft_consensus.cc:686] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: 517e9795228541b58fc43b37dea6a4c7, State: Running, Role: LEADER
I20260807 18:27:42.821995 28754 consensus_queue.cc:237] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40231 } } peers { permanent_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45173 } } peers { permanent_uuid: "517e9795228541b58fc43b37dea6a4c7" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 36381 } }
I20260807 18:27:42.827394 28245 catalog_manager.cc:5495] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 517e9795228541b58fc43b37dea6a4c7 (127.27.1.194). New cstate: current_term: 1 leader_uuid: "517e9795228541b58fc43b37dea6a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f535e143b41e4725b27fccf6d7c9d4dc" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac496eb18224aa594f9f9bc74fac5d6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "517e9795228541b58fc43b37dea6a4c7" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 36381 } health_report { overall_health: HEALTHY } } }
Aug 07 18:27:42 dist-test-slave-1xrh krb5kdc[28200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:27:42.850281 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.834232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.27.1.193:40231)
0807 18:27:42.834407 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.834412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.835079 (+ 667us) server_negotiation.cc:407] Connection header received
0807 18:27:42.835284 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.835288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.835380 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.835472 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:42.837013 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.837849 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.839113 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.839692 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.839802 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.843894 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:42.843910 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:42.843927 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:42.843974 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:42.847407 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.848109 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.848117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.848120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.848266 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.848672 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.848676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.848678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.849151 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:42.849435 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.849659 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.849867 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 18:27:42.855029 28447 tablet.cc:2369] T e33d3e11b08c487bb2470cc6f341e653 P f535e143b41e4725b27fccf6d7c9d4dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:27:42.876258 28585 tablet.cc:2369] T e33d3e11b08c487bb2470cc6f341e653 P 517e9795228541b58fc43b37dea6a4c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 18:27:42 dist-test-slave-1xrh krb5kdc[28200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260807 18:27:42.885974 28730 tablet.cc:2369] T e33d3e11b08c487bb2470cc6f341e653 P 7ac496eb18224aa594f9f9bc74fac5d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 18:27:42 dist-test-slave-1xrh krb5kdc[28200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:27:42.902412 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.888025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.27.1.193:40231)
0807 18:27:42.888310 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.888320 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.888671 (+ 351us) server_negotiation.cc:407] Connection header received
0807 18:27:42.888895 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.888900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.888965 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.889079 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:42.890429 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.891312 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.892591 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.893031 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.893174 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.897243 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:42.897248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:42.897252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:42.897288 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:42.900075 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.900671 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.900675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.900678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.900803 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.901199 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.901202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.901204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.901460 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:42.901677 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.901960 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.902146 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Aug 07 18:27:42 dist-test-slave-1xrh krb5kdc[28200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 18:27:42 dist-test-slave-1xrh krb5kdc[28200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:27:42.955339 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.940084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.27.1.193:40231)
0807 18:27:42.940339 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.940348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.940660 (+ 312us) server_negotiation.cc:407] Connection header received
0807 18:27:42.940878 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.940882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.940950 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.941077 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:42.942611 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.943366 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.944651 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.945243 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.945389 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.949460 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:42.949465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:42.949468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:42.949513 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:42.952426 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.953111 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.953117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.953121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.953250 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.953666 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.953671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.953675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.954041 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:42.954316 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.954669 (+ 353us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.954919 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 18:27:42.957346 28360 tablet_service.cc:2908] Rejecting scan request for tablet e33d3e11b08c487bb2470cc6f341e653: Uninitialized: clean time has not yet been initialized
I20260807 18:27:42.960862 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 28308
I20260807 18:27:42.973371 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:42.961674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.27.1.193:40231)
0807 18:27:42.961829 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:42.961833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:42.962237 (+ 404us) server_negotiation.cc:407] Connection header received
0807 18:27:42.962371 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:42.962374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:42.962429 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:42.962503 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:42.963749 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.964447 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.965597 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:42.965998 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:42.966101 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:42.968392 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:42.968397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:42.968400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:42.968432 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:42.971005 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.971668 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.971672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.971674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.971790 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:42.972146 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:42.972149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:42.972151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:42.972419 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:42.972626 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:42.972961 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 18:27:42.973133 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 18:27:42.974118 28441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:27:43.063803 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 28308
I20260807 18:27:43.086174 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 28452
Aug 07 18:27:43 dist-test-slave-1xrh krb5kdc[28200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:27:43.101540 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:43.087039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.27.1.194:36381)
0807 18:27:43.087281 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:43.087287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:43.087309 (+ 22us) server_negotiation.cc:407] Connection header received
0807 18:27:43.087517 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:43.087522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:43.087589 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:43.087698 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:43.088903 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:43.089647 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:43.090907 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:43.091541 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:43.091684 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:43.095945 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:43.095959 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:43.095975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:43.096026 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:43.099000 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:43.099680 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:43.099687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:43.099689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:43.099881 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:43.100212 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:43.100215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:43.100217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:43.100569 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:43.100795 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:43.100990 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 18:27:43.101186 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:27:43.102245 28579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:27:43.194304 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 28452
I20260807 18:27:43.215183 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 28590
Aug 07 18:27:43 dist-test-slave-1xrh krb5kdc[28200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:27:43.230710 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:43.216165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.27.1.195:45173)
0807 18:27:43.216375 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:43.216380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:43.216396 (+ 16us) server_negotiation.cc:407] Connection header received
0807 18:27:43.216449 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:43.216453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:43.216526 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:43.216640 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:43.217913 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:43.218652 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:43.220217 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:43.220624 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:43.220763 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:43.224987 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:43.225002 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:43.225019 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:43.225052 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:43.227876 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:43.228560 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:43.228568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:43.228570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:43.228724 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:43.229126 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:43.229131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:43.229135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:43.229558 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:43.229852 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:43.230151 (+ 299us) server_negotiation.cc:299] Negotiation successful
0807 18:27:43.230388 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 18:27:43.231436 28724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:27:43.322736 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 28590
I20260807 18:27:43.343592 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 28214
I20260807 18:27:43.344843 28277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:27:43.454825 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 28214
2026-08-07T18:27:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4248 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:27:43 dist-test-slave-1xrh krb5kdc[28778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:27:43 dist-test-slave-1xrh krb5kdc[28778](info): set up 2 sockets
Aug 07 18:27:43 dist-test-slave-1xrh krb5kdc[28778](info): commencing operation
krb5kdc: starting...
W20260807 18:27:44.568675 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:27:44 dist-test-slave-1xrh krb5kdc[28778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:27:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:27:44.770270 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:42065
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:42065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:45.094300 28791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:45.094590 28791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:45.094681 28791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:45.101891 28791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:27:45.102020 28791 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 18:27:45.102097 28791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:45.102203 28791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:45.102288 28791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:27:45.102375 28791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:27:45.112299 28791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:42065
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:42065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:45.113585 28791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:45.114918 28791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:45.126911 28796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:45.126863 28799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:45.126827 28797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:45.127983 28791 server_base.cc:1034] running on GCE node
I20260807 18:27:45.128736 28791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:45.130198 28791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:45.131460 28791 hybrid_clock.cc:648] HybridClock initialized: now 1786127265131379 us; error 54 us; skew 500 ppm
I20260807 18:27:45.131800 28791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:45 dist-test-slave-1xrh krb5kdc[28778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127265, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:45.134815 28791 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:27:45.136333 28791 webserver.cc:460] Webserver started at http://127.27.1.254:36241/ using document root <none> and password file <none>
I20260807 18:27:45.137037 28791 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:45.137171 28791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:45.137514 28791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:45.140524 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "8d1b280a013e4af1920e1a4495bedbc7"
format_stamp: "Formatted at 2026-08-07 18:27:45 on dist-test-slave-1xrh"
server_key: "0d41f04de667c1f31bedb0fd683a8bb0"
server_key_iv: "f53da68613521cf3ecbc5388e6657fd6"
server_key_version: "encryptionkey@0"
I20260807 18:27:45.141340 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "8d1b280a013e4af1920e1a4495bedbc7"
format_stamp: "Formatted at 2026-08-07 18:27:45 on dist-test-slave-1xrh"
server_key: "0d41f04de667c1f31bedb0fd683a8bb0"
server_key_iv: "f53da68613521cf3ecbc5388e6657fd6"
server_key_version: "encryptionkey@0"
I20260807 18:27:45.146384 28791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 18:27:45.149750 28807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:45.150990 28791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:27:45.151216 28791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "8d1b280a013e4af1920e1a4495bedbc7"
format_stamp: "Formatted at 2026-08-07 18:27:45 on dist-test-slave-1xrh"
server_key: "0d41f04de667c1f31bedb0fd683a8bb0"
server_key_iv: "f53da68613521cf3ecbc5388e6657fd6"
server_key_version: "encryptionkey@0"
I20260807 18:27:45.151428 28791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:45.173990 28791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:45.177796 28791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:45.178068 28791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:45.200533 28791 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:42065
I20260807 18:27:45.200555 28871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:42065 every 8 connection(s)
I20260807 18:27:45.201996 28791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:27:45.206059 28872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:45.208967 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 28791
I20260807 18:27:45.209213 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:27:45.209882 27655 external_mini_cluster.cc:1428] Setting key 276bda67cc4debd931c79ad74210a19a
I20260807 18:27:45.217773 28872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7: Bootstrap starting.
I20260807 18:27:45.220796 28872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:45.221757 28872 log.cc:824] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7: Log is configured to *not* fsync() on all Append() calls
Aug 07 18:27:45 dist-test-slave-1xrh krb5kdc[28778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:45.224977 28872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7: No bootstrap required, opened a new log
I20260807 18:27:45.232707 28875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:45.211995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.27.1.254:42065)
0807 18:27:45.212730 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:45.212741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:45.212780 (+ 39us) server_negotiation.cc:407] Connection header received
0807 18:27:45.214460 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:45.214483 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:45.214898 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:45.215315 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:45.216719 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:45.218299 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:45.219600 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:45.220286 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:45.220490 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:45.225478 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:45.225485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:45.225502 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:45.225543 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:45.228743 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:45.229464 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:45.229470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:45.229476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:45.229644 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:45.230063 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:45.230067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:45.230068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:45.230610 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:45.230927 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:45.231353 (+ 426us) server_negotiation.cc:299] Negotiation successful
0807 18:27:45.231743 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":335,"threads_started":1}
I20260807 18:27:45.232715 28872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1b280a013e4af1920e1a4495bedbc7" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42065 } }
I20260807 18:27:45.233259 28872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:45.233409 28872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1b280a013e4af1920e1a4495bedbc7, State: Initialized, Role: FOLLOWER
I20260807 18:27:45.234012 28872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1b280a013e4af1920e1a4495bedbc7" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42065 } }
I20260807 18:27:45.234285 28872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:27:45.234444 28872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:27:45.234618 28872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:45.237373 28872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1b280a013e4af1920e1a4495bedbc7" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42065 } }
I20260807 18:27:45.237880 28872 leader_election.cc:302] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d1b280a013e4af1920e1a4495bedbc7; no voters:
I20260807 18:27:45.238606 28872 leader_election.cc:288] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:27:45.238715 28877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:45.241122 28872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:27:45.241509 28877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [term 1 LEADER]: Becoming Leader. State: Replica: 8d1b280a013e4af1920e1a4495bedbc7, State: Running, Role: LEADER
I20260807 18:27:45.242117 28877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1b280a013e4af1920e1a4495bedbc7" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42065 } }
I20260807 18:27:45.245898 28879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1b280a013e4af1920e1a4495bedbc7. Latest consensus state: current_term: 1 leader_uuid: "8d1b280a013e4af1920e1a4495bedbc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1b280a013e4af1920e1a4495bedbc7" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42065 } } }
I20260807 18:27:45.245985 28878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d1b280a013e4af1920e1a4495bedbc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1b280a013e4af1920e1a4495bedbc7" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42065 } } }
I20260807 18:27:45.246289 28879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [sys.catalog]: This master's current role is: LEADER
I20260807 18:27:45.246372 28878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7 [sys.catalog]: This master's current role is: LEADER
I20260807 18:27:45.248035 28888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:27:45.253093 28888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:27:45.261358 28888 catalog_manager.cc:1347] Generated new cluster ID: b35bcebe5ad540ab9d5ac8e97e4dc67f
I20260807 18:27:45.261483 28888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:27:45.284574 28888 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:27:45.285948 28888 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:27:45.294410 28888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7: Generated new TSK 0
I20260807 18:27:45.295261 28888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:27:45.354240 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42065
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:45.723539 28898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:45.723872 28898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:45.724035 28898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:45.731094 28898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:45.731298 28898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:45.731443 28898 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 18:27:45.731580 28898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:27:45.741585 28898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.1.254:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:45.742944 28898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:45.744506 28898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:45.757639 28903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:45.757707 28906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:45.758164 28904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:45.758605 28898 server_base.cc:1034] running on GCE node
I20260807 18:27:45.759238 28898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:45.760110 28898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:45.761329 28898 hybrid_clock.cc:648] HybridClock initialized: now 1786127265761256 us; error 50 us; skew 500 ppm
I20260807 18:27:45.761622 28898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:45 dist-test-slave-1xrh krb5kdc[28778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127265, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:45.764909 28898 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:27:45.766263 28898 webserver.cc:460] Webserver started at http://127.27.1.193:41753/ using document root <none> and password file <none>
I20260807 18:27:45.766981 28898 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:45.767066 28898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:45.767401 28898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:45.770145 28898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "40137d2ca6e6489fa438cfcd710129fd"
format_stamp: "Formatted at 2026-08-07 18:27:45 on dist-test-slave-1xrh"
server_key: "687b2f558c508225c87a82779af74314"
server_key_iv: "4bc9474bafc633b8d443d3d7ee4f0a5a"
server_key_version: "encryptionkey@0"
I20260807 18:27:45.770910 28898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "40137d2ca6e6489fa438cfcd710129fd"
format_stamp: "Formatted at 2026-08-07 18:27:45 on dist-test-slave-1xrh"
server_key: "687b2f558c508225c87a82779af74314"
server_key_iv: "4bc9474bafc633b8d443d3d7ee4f0a5a"
server_key_version: "encryptionkey@0"
I20260807 18:27:45.775817 28898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260807 18:27:45.779249 28913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:45.780578 28898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 18:27:45.780751 28898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "40137d2ca6e6489fa438cfcd710129fd"
format_stamp: "Formatted at 2026-08-07 18:27:45 on dist-test-slave-1xrh"
server_key: "687b2f558c508225c87a82779af74314"
server_key_iv: "4bc9474bafc633b8d443d3d7ee4f0a5a"
server_key_version: "encryptionkey@0"
I20260807 18:27:45.780972 28898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:45.799979 28898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:45.804073 28898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:45.804360 28898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:45.807219 28898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:45.807327 28898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:45.807493 28898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:45.807578 28898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 18:27:45 dist-test-slave-1xrh krb5kdc[28778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127265, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:45.830907 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:45.807796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:40033 (local address 127.27.1.254:42065)
0807 18:27:45.808513 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:45.808518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:45.809405 (+ 887us) server_negotiation.cc:407] Connection header received
0807 18:27:45.811491 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:45.811496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:45.811586 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:45.811718 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:45.813750 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:45.815063 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:45.817700 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:45.818253 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:45.818403 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:45.823676 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:45.823684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:45.823691 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:45.823745 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:45.827501 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:45.828549 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:45.828556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:45.828563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:45.828707 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:45.829170 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:45.829175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:45.829179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:45.829454 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:45.829615 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:45.830310 (+ 695us) server_negotiation.cc:299] Negotiation successful
0807 18:27:45.830539 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":476,"threads_started":1}
I20260807 18:27:45.832329 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:45.808430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42065 (local address 127.27.1.193:40033)
0807 18:27:45.809252 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:45.809292 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:45.811227 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:45.811932 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:45.811940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:45.812672 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:45.813501 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:45.813524 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:45.815286 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:45.815294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:45.817539 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:45.817555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:45.818515 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:45.818527 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:45.818966 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:45.819851 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:45.819908 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:45.823410 (+ 3502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:45.827683 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:45.827695 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:45.827710 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:45.828385 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:45.828824 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:45.828830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:45.828833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:45.829050 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:45.829587 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:45.829595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:45.829941 (+ 346us) client_negotiation.cc:769] Sending connection context
0807 18:27:45.830213 (+ 272us) client_negotiation.cc:240] Negotiation successful
0807 18:27:45.830724 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":365,"threads_started":1}
I20260807 18:27:45.854408 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:45.838482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42065 (local address 127.27.1.193:60979)
0807 18:27:45.839100 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:45.839116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:45.839310 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:45.839631 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:45.839635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:45.840096 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:45.840376 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:45.840384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:45.841455 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:45.841462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:45.843927 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:45.843949 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:45.845283 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:45.845292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:45.845540 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:45.846257 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:45.846288 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:45.847436 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:45.851702 (+ 4266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:45.851712 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:45.851717 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:45.852160 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:45.852621 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:45.852624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:45.852626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:45.852740 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:45.853475 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:45.853478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:45.853630 (+ 152us) client_negotiation.cc:769] Sending connection context
0807 18:27:45.853748 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 18:27:45.854056 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":349,"threads_started":1}
I20260807 18:27:45.854815 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:45.838374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:60979 (local address 127.27.1.254:42065)
0807 18:27:45.838537 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:45.838541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:45.839239 (+ 698us) server_negotiation.cc:407] Connection header received
0807 18:27:45.839424 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:45.839429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:45.839503 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:45.839604 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:45.840529 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:45.841300 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:45.844171 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:45.845029 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:45.845186 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:45.847689 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:45.847697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:45.847706 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:45.847751 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:45.851550 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:45.852295 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:45.852303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:45.852310 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:45.852496 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:45.852872 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:45.852880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:45.852885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:45.853325 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:45.853538 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:45.853953 (+ 415us) server_negotiation.cc:299] Negotiation successful
0807 18:27:45.854280 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 18:27:45.861656 28898 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:33365
I20260807 18:27:45.861686 29038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:33365 every 8 connection(s)
I20260807 18:27:45.862879 28898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:27:45.864977 29039 heartbeater.cc:347] Connected to a master server at 127.27.1.254:42065
I20260807 18:27:45.865216 29039 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:45.865852 29039 heartbeater.cc:511] Master 127.27.1.254:42065 requested a full tablet report, sending...
I20260807 18:27:45.867229 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 28898
I20260807 18:27:45.867452 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:27:45.868018 28826 ts_manager.cc:194] Registered new tserver with Master: 40137d2ca6e6489fa438cfcd710129fd (127.27.1.193:33365)
I20260807 18:27:45.868110 27655 external_mini_cluster.cc:1428] Setting key 4251057fa67aa80fe250a85db0dd693e
I20260807 18:27:45.869684 28826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:40033
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:27:45.912170 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42065
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:46.245486 29043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:46.245772 29043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:46.245985 29043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:46.254215 29043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:46.254369 29043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:46.254508 29043 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 18:27:46.254608 29043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:27:46.265643 29043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.1.254:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:46.266919 29043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:46.268493 29043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:46.282693 29051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:46.282666 29048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:46.282809 29049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:46.283833 29043 server_base.cc:1034] running on GCE node
I20260807 18:27:46.284403 29043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:46.285389 29043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:46.286633 29043 hybrid_clock.cc:648] HybridClock initialized: now 1786127266286578 us; error 51 us; skew 500 ppm
I20260807 18:27:46.286944 29043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:46 dist-test-slave-1xrh krb5kdc[28778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127266, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:46.290057 29043 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:27:46.291657 29043 webserver.cc:460] Webserver started at http://127.27.1.194:37323/ using document root <none> and password file <none>
I20260807 18:27:46.292425 29043 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:46.292557 29043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:46.292902 29043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:46.296100 29043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "1269152c209c4e00920666da1344783f"
format_stamp: "Formatted at 2026-08-07 18:27:46 on dist-test-slave-1xrh"
server_key: "ed64ae6317c29ef4ee48d7f7f6d2d78f"
server_key_iv: "6f13f7e4b753622e735844f2ee8b763e"
server_key_version: "encryptionkey@0"
I20260807 18:27:46.296926 29043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "1269152c209c4e00920666da1344783f"
format_stamp: "Formatted at 2026-08-07 18:27:46 on dist-test-slave-1xrh"
server_key: "ed64ae6317c29ef4ee48d7f7f6d2d78f"
server_key_iv: "6f13f7e4b753622e735844f2ee8b763e"
server_key_version: "encryptionkey@0"
I20260807 18:27:46.302037 29043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:27:46.305498 29058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:46.306769 29043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:27:46.306952 29043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "1269152c209c4e00920666da1344783f"
format_stamp: "Formatted at 2026-08-07 18:27:46 on dist-test-slave-1xrh"
server_key: "ed64ae6317c29ef4ee48d7f7f6d2d78f"
server_key_iv: "6f13f7e4b753622e735844f2ee8b763e"
server_key_version: "encryptionkey@0"
I20260807 18:27:46.307204 29043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:46.337302 29043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:46.341457 29043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:46.341732 29043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:46.344748 29043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:46.344856 29043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:46.345031 29043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:46.345110 29043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 18:27:46 dist-test-slave-1xrh krb5kdc[28778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127266, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:46.368649 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:46.345532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:38115 (local address 127.27.1.254:42065)
0807 18:27:46.345740 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:46.345748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:46.347484 (+ 1736us) server_negotiation.cc:407] Connection header received
0807 18:27:46.349643 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:46.349651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:46.349760 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:46.349887 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:46.351913 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.352933 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.355601 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.356340 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.356493 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:46.360817 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:46.360822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:46.360825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:46.360860 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:46.364816 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:46.365822 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:46.365830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:46.365835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:46.365993 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:46.366621 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:46.366625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:46.366627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:46.366924 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:46.367110 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:46.368057 (+ 947us) server_negotiation.cc:299] Negotiation successful
0807 18:27:46.368281 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:27:46.369918 29067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:46.346223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42065 (local address 127.27.1.194:38115)
0807 18:27:46.347326 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:46.347364 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:46.349233 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:46.350155 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:46.350162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:46.350834 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:46.351723 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:46.351743 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.353084 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.353091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:46.355384 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:46.355413 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.356610 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.356618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:46.356821 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:46.357504 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:46.357526 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:46.360575 (+ 3049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:46.365026 (+ 4451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:46.365038 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:46.365059 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:46.365648 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:46.366111 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:46.366116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:46.366120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:46.366441 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:46.367081 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:46.367092 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:46.367629 (+ 537us) client_negotiation.cc:769] Sending connection context
0807 18:27:46.367947 (+ 318us) client_negotiation.cc:240] Negotiation successful
0807 18:27:46.368467 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":558,"threads_started":1}
I20260807 18:27:46.394320 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:46.377211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42065 (local address 127.27.1.194:60565)
0807 18:27:46.378073 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:46.378102 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:46.378293 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:46.378660 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:46.378668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:46.379482 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:46.380048 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:46.380071 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.381324 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.381333 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:46.383818 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:46.383859 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.384860 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.384871 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:46.385174 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:46.386044 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:46.386075 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:46.387291 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:46.391277 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:46.391282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:46.391286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:46.391872 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:46.392441 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:46.392445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:46.392447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:46.392563 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:46.393481 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:46.393484 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:46.393643 (+ 159us) client_negotiation.cc:769] Sending connection context
0807 18:27:46.393766 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 18:27:46.394001 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":493,"threads_started":1}
I20260807 18:27:46.394807 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:46.377202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:60565 (local address 127.27.1.254:42065)
0807 18:27:46.377419 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:46.377429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:46.378205 (+ 776us) server_negotiation.cc:407] Connection header received
0807 18:27:46.378443 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:46.378447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:46.378503 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:46.378582 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:46.380210 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.381036 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.384043 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.384619 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.384759 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:46.387549 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:46.387558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:46.387583 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:46.387626 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:46.391058 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:46.392050 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:46.392060 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:46.392067 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:46.392309 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:46.392691 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:46.392700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:46.392707 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:46.393327 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:46.393562 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:46.393952 (+ 390us) server_negotiation.cc:299] Negotiation successful
0807 18:27:46.394269 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 18:27:46.404096 29043 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:34945
I20260807 18:27:46.404124 29182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:34945 every 8 connection(s)
I20260807 18:27:46.405463 29043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:27:46.407646 29183 heartbeater.cc:347] Connected to a master server at 127.27.1.254:42065
I20260807 18:27:46.407836 29183 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:46.408406 29183 heartbeater.cc:511] Master 127.27.1.254:42065 requested a full tablet report, sending...
I20260807 18:27:46.409752 28826 ts_manager.cc:194] Registered new tserver with Master: 1269152c209c4e00920666da1344783f (127.27.1.194:34945)
I20260807 18:27:46.410960 28826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:38115
I20260807 18:27:46.411614 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 29043
I20260807 18:27:46.411895 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:27:46.412462 27655 external_mini_cluster.cc:1428] Setting key c74e84493de8b4dec462fddddcf8fda5
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:27:46.458303 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42065
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:46.785990 29187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:46.786260 29187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:46.786355 29187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:46.794063 29187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:46.794166 29187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:46.794250 29187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 18:27:46.794329 29187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:27:46.803864 29187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.1.254:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:46.805056 29187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:46.806377 29187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:46.819336 29193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:46.819474 29194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:46.819480 29196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:46.820051 29187 server_base.cc:1034] running on GCE node
I20260807 18:27:46.820641 29187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:46.821565 29187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:46.822808 29187 hybrid_clock.cc:648] HybridClock initialized: now 1786127266822732 us; error 55 us; skew 500 ppm
I20260807 18:27:46.823109 29187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:46 dist-test-slave-1xrh krb5kdc[28778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127266, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:46.826269 29187 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:27:46.827636 29187 webserver.cc:460] Webserver started at http://127.27.1.195:35047/ using document root <none> and password file <none>
I20260807 18:27:46.828263 29187 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:46.828349 29187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:46.828614 29187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:46.831414 29187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "023df7f0b0f54dcab50420358937f62d"
format_stamp: "Formatted at 2026-08-07 18:27:46 on dist-test-slave-1xrh"
server_key: "89f6e30190926702b4f75ab58e05cda3"
server_key_iv: "d7a865d8fd1780b4e6dd3782bb6018ac"
server_key_version: "encryptionkey@0"
I20260807 18:27:46.832203 29187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "023df7f0b0f54dcab50420358937f62d"
format_stamp: "Formatted at 2026-08-07 18:27:46 on dist-test-slave-1xrh"
server_key: "89f6e30190926702b4f75ab58e05cda3"
server_key_iv: "d7a865d8fd1780b4e6dd3782bb6018ac"
server_key_version: "encryptionkey@0"
I20260807 18:27:46.837702 29187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:27:46.840936 29203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:46.842352 29187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:27:46.842573 29187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "023df7f0b0f54dcab50420358937f62d"
format_stamp: "Formatted at 2026-08-07 18:27:46 on dist-test-slave-1xrh"
server_key: "89f6e30190926702b4f75ab58e05cda3"
server_key_iv: "d7a865d8fd1780b4e6dd3782bb6018ac"
server_key_version: "encryptionkey@0"
I20260807 18:27:46.842808 29187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:46.859761 29187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:46.863937 29187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:46.864207 29187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:46.867203 29187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:46.867309 29187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260807 18:27:46.867484 29187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:46.867574 29187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:46.872701 29039 heartbeater.cc:503] Master 127.27.1.254:42065 was elected leader, sending a full tablet report...
Aug 07 18:27:46 dist-test-slave-1xrh krb5kdc[28778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127266, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:46.892848 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:46.867958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:43347 (local address 127.27.1.254:42065)
0807 18:27:46.868168 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:46.868174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:46.869539 (+ 1365us) server_negotiation.cc:407] Connection header received
0807 18:27:46.871575 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:46.871581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:46.871684 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:46.871828 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:46.873883 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.875063 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.878608 (+ 3545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.879350 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.879492 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:46.885283 (+ 5791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:46.885290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:46.885297 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:46.885346 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:46.888724 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:46.889911 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:46.889918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:46.889924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:46.890125 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:46.890853 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:46.890860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:46.890864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:46.891296 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:46.891508 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:46.892230 (+ 722us) server_negotiation.cc:299] Negotiation successful
0807 18:27:46.892483 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 18:27:46.893996 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:46.868399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42065 (local address 127.27.1.195:43347)
0807 18:27:46.869355 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:46.869404 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:46.871287 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:46.872154 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:46.872165 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:46.872731 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:46.873662 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:46.873696 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.875508 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.875516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:46.878425 (+ 2909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:46.878451 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.879623 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.879633 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:46.879915 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:46.880847 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:46.880891 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:46.885009 (+ 4118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:46.888903 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:46.888913 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:46.888933 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:46.889769 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:46.890291 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:46.890302 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:46.890309 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:46.890697 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:46.891455 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:46.891464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:46.891842 (+ 378us) client_negotiation.cc:769] Sending connection context
0807 18:27:46.892123 (+ 281us) client_negotiation.cc:240] Negotiation successful
0807 18:27:46.892672 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":422,"threads_started":1}
I20260807 18:27:46.917631 29187 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:39883
I20260807 18:27:46.917688 29326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:39883 every 8 connection(s)
I20260807 18:27:46.918931 29187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:27:46.919346 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:46.903062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:59279 (local address 127.27.1.254:42065)
0807 18:27:46.903306 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:46.903312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:46.904231 (+ 919us) server_negotiation.cc:407] Connection header received
0807 18:27:46.904437 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:46.904445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:46.904546 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:46.904652 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:46.906344 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.907357 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.909096 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.909893 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.910063 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:46.912639 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:46.912649 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:46.912657 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:46.912710 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:46.916215 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:46.917114 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:46.917119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:46.917122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:46.917292 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:46.917827 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:46.917832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:46.917835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:46.918141 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:46.918302 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:46.918710 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 18:27:46.918945 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260807 18:27:46.919464 29288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:46.903120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42065 (local address 127.27.1.195:59279)
0807 18:27:46.904127 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:46.904145 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:46.904308 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:46.904720 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:46.904731 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:46.905523 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:46.906145 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:46.906166 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.907557 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.907564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:46.908934 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:46.908947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.910123 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.910134 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:46.910331 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:46.911186 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:46.911221 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:46.912408 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:46.916363 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:46.916368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:46.916376 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:46.916955 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:46.917457 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:46.917464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:46.917469 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:46.917657 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:46.918267 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:46.918272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:46.918463 (+ 191us) client_negotiation.cc:769] Sending connection context
0807 18:27:46.918621 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 18:27:46.918947 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":562,"threads_started":1}
I20260807 18:27:46.922688 29327 heartbeater.cc:347] Connected to a master server at 127.27.1.254:42065
I20260807 18:27:46.922924 29327 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:46.923645 29327 heartbeater.cc:511] Master 127.27.1.254:42065 requested a full tablet report, sending...
I20260807 18:27:46.924958 28826 ts_manager.cc:194] Registered new tserver with Master: 023df7f0b0f54dcab50420358937f62d (127.27.1.195:39883)
I20260807 18:27:46.925755 28826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:43347
I20260807 18:27:46.927703 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 29187
I20260807 18:27:46.927909 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:27:46.928339 27655 external_mini_cluster.cc:1428] Setting key a3dcc92bbab84d289edd709fa42fe789
I20260807 18:27:46.931272 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 18:27:46 dist-test-slave-1xrh krb5kdc[28778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127266, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 18:27:46 dist-test-slave-1xrh krb5kdc[28778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127266, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:46.982201 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:46.968099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.27.1.254:42065)
0807 18:27:46.968364 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:46.968372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:46.968670 (+ 298us) server_negotiation.cc:407] Connection header received
0807 18:27:46.968948 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:46.968953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:46.969021 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:46.969109 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:46.970499 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.971292 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.972570 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:46.973045 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:46.973138 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:46.976952 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:46.976957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:46.976960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:46.976995 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:46.979642 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:46.980268 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:46.980275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:46.980279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:46.980405 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:46.980781 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:46.980786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:46.980790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:46.981063 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:46.981309 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:46.981587 (+ 278us) server_negotiation.cc:299] Negotiation successful
0807 18:27:46.981810 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260807 18:27:46.986325 28826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:27:46.993084 28826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:27:47.018589 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.005502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.27.1.195:39883)
0807 18:27:47.010181 (+ 4679us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.010191 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.010221 (+ 30us) server_negotiation.cc:407] Connection header received
0807 18:27:47.010341 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.010356 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.010562 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.011074 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:47.011722 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.014022 (+ 2300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.016377 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.017288 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.017748 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.017992 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.018066 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.018234 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4523,"thread_start_us":4334,"threads_started":1}
I20260807 18:27:47.018638 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.007686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.27.1.193:33365)
0807 18:27:47.009447 (+ 1761us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.009459 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.009507 (+ 48us) server_negotiation.cc:407] Connection header received
0807 18:27:47.009628 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.009641 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.009936 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.011116 (+ 1180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:47.011430 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.013077 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.015748 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.017243 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.017391 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.017598 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.018151 (+ 553us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.018320 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1560,"thread_start_us":1396,"threads_started":1}
I20260807 18:27:47.019003 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.005752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.27.1.194:34945)
0807 18:27:47.009765 (+ 4013us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.009777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.009806 (+ 29us) server_negotiation.cc:407] Connection header received
0807 18:27:47.009936 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.009944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.010158 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.011424 (+ 1266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:47.011851 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.012866 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.015858 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.017243 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.017391 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.017600 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.018257 (+ 657us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.018436 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3849,"thread_start_us":3723,"threads_started":1}
I20260807 18:27:47.019462 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.007091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:33365 (local address 127.0.0.1:47730)
0807 18:27:47.008712 (+ 1621us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:47.008734 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:47.008870 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:47.010209 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:47.010216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:47.010255 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:47.010753 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.010766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.013231 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.013238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.015582 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.015598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.017451 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.017460 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.017711 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.017873 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 18:27:47.018018 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 18:27:47.019117 (+ 1099us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1471,"mutex_wait_us":1095,"thread_start_us":1276,"threads_started":1}
I20260807 18:27:47.020044 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.005227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:39883 (local address 127.0.0.1:35242)
0807 18:27:47.006715 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:47.006760 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:47.006936 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:47.010909 (+ 3973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:47.010915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:47.010969 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:47.011515 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.011528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.014203 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.014208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.016233 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.016244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.017529 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.017534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.017843 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.017877 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 18:27:47.017997 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 18:27:47.019689 (+ 1692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"mutex_wait_us":775,"thread_start_us":2039,"threads_started":1}
I20260807 18:27:47.020190 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.005329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:34945 (local address 127.0.0.1:35408)
0807 18:27:47.007573 (+ 2244us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:47.007591 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:47.007701 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:47.010463 (+ 2762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:47.010468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:47.010495 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:47.011676 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.011694 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.012999 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.013006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.015700 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.015715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.017852 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.017858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.018025 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.018057 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 18:27:47.018138 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 18:27:47.019779 (+ 1641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2106,"mutex_wait_us":1521,"thread_start_us":1015,"threads_started":1}
I20260807 18:27:47.022965 28965 tablet_service.cc:1467] Processing CreateTablet for tablet 19f701b1d4e741808b2bd747b475f6ee (DEFAULT_TABLE table=test-table [id=65e5d4339c26480ea68bffec3d00c3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:47.023451 29112 tablet_service.cc:1467] Processing CreateTablet for tablet 19f701b1d4e741808b2bd747b475f6ee (DEFAULT_TABLE table=test-table [id=65e5d4339c26480ea68bffec3d00c3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:47.023451 29254 tablet_service.cc:1467] Processing CreateTablet for tablet 19f701b1d4e741808b2bd747b475f6ee (DEFAULT_TABLE table=test-table [id=65e5d4339c26480ea68bffec3d00c3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:47.024492 28965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f701b1d4e741808b2bd747b475f6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:47.024806 29112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f701b1d4e741808b2bd747b475f6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:47.024801 29254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f701b1d4e741808b2bd747b475f6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:47.035777 29346 tablet_bootstrap.cc:492] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: Bootstrap starting.
I20260807 18:27:47.039237 29348 tablet_bootstrap.cc:492] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: Bootstrap starting.
I20260807 18:27:47.040302 29346 tablet_bootstrap.cc:654] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:47.041700 29346 log.cc:824] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:47.042757 29348 tablet_bootstrap.cc:654] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:47.043267 29347 tablet_bootstrap.cc:492] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: Bootstrap starting.
I20260807 18:27:47.044144 29348 log.cc:824] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:47.045550 29346 tablet_bootstrap.cc:492] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: No bootstrap required, opened a new log
I20260807 18:27:47.045806 29346 ts_tablet_manager.cc:1397] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260807 18:27:47.046981 29347 tablet_bootstrap.cc:654] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:47.048079 29347 log.cc:824] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:47.048105 29348 tablet_bootstrap.cc:492] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: No bootstrap required, opened a new log
I20260807 18:27:47.048391 29348 ts_tablet_manager.cc:1397] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 18:27:47.050518 29347 tablet_bootstrap.cc:492] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: No bootstrap required, opened a new log
I20260807 18:27:47.050777 29347 ts_tablet_manager.cc:1397] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260807 18:27:47.056735 29346 raft_consensus.cc:348] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } }
I20260807 18:27:47.057201 29346 raft_consensus.cc:374] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:47.057312 29346 raft_consensus.cc:729] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40137d2ca6e6489fa438cfcd710129fd, State: Initialized, Role: FOLLOWER
I20260807 18:27:47.057943 29346 consensus_queue.cc:260] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } }
I20260807 18:27:47.058596 29347 raft_consensus.cc:348] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } }
I20260807 18:27:47.059018 29347 raft_consensus.cc:374] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:47.059149 29347 raft_consensus.cc:729] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023df7f0b0f54dcab50420358937f62d, State: Initialized, Role: FOLLOWER
I20260807 18:27:47.059181 29348 raft_consensus.cc:348] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } }
I20260807 18:27:47.059588 29348 raft_consensus.cc:374] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:47.059700 29348 raft_consensus.cc:729] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1269152c209c4e00920666da1344783f, State: Initialized, Role: FOLLOWER
I20260807 18:27:47.059948 29346 ts_tablet_manager.cc:1428] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260807 18:27:47.059695 29347 consensus_queue.cc:260] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } }
I20260807 18:27:47.060276 29348 consensus_queue.cc:260] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } }
I20260807 18:27:47.062065 29183 heartbeater.cc:503] Master 127.27.1.254:42065 was elected leader, sending a full tablet report...
I20260807 18:27:47.062700 29348 ts_tablet_manager.cc:1428] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260807 18:27:47.063400 29347 ts_tablet_manager.cc:1428] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260807 18:27:47.063423 29327 heartbeater.cc:503] Master 127.27.1.254:42065 was elected leader, sending a full tablet report...
W20260807 18:27:47.115392 29040 tablet.cc:2369] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:27:47.157720 29184 tablet.cc:2369] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:27:47.172032 29328 tablet.cc:2369] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:27:47.274470 29354 raft_consensus.cc:482] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:47.274895 29354 raft_consensus.cc:504] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } }
I20260807 18:27:47.277494 29354 leader_election.cc:288] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 023df7f0b0f54dcab50420358937f62d (127.27.1.195:39883), 40137d2ca6e6489fa438cfcd710129fd (127.27.1.193:33365)
I20260807 18:27:47.286784 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.278482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:60061 (local address 127.27.1.195:39883)
0807 18:27:47.278724 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.278733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.279911 (+ 1178us) server_negotiation.cc:407] Connection header received
0807 18:27:47.280131 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.280135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.280216 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.280303 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:47.281383 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.282353 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.284549 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.285600 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.285773 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.285847 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.286361 (+ 514us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.286503 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 18:27:47.287079 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.277718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:39883 (local address 127.27.1.194:60061)
0807 18:27:47.279786 (+ 2068us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:47.279801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:47.279975 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:47.280450 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:47.280458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:47.280491 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:47.281166 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.281180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.282498 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.282507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.284318 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.284331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.285884 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.285893 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.286142 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.286196 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 18:27:47.286315 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 18:27:47.286492 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1931,"mutex_wait_us":234,"thread_start_us":1436,"threads_started":1}
I20260807 18:27:47.288264 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.278208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:35351 (local address 127.27.1.193:33365)
0807 18:27:47.278473 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.278478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.279611 (+ 1133us) server_negotiation.cc:407] Connection header received
0807 18:27:47.280014 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.280022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.280132 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.280233 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:47.280877 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.282022 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.285296 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.286349 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.286515 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.286584 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.287744 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.287868 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 18:27:47.288323 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.277718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:33365 (local address 127.27.1.194:35351)
0807 18:27:47.279471 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:47.279493 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:47.279765 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:47.280263 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:47.280271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:47.280299 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:47.280716 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.280727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.282189 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.282197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.285096 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.285115 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.287371 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.287382 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.287568 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.287606 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 18:27:47.287718 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 18:27:47.287833 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1557,"mutex_wait_us":77,"thread_start_us":1224,"threads_started":1}
I20260807 18:27:47.288200 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f701b1d4e741808b2bd747b475f6ee" candidate_uuid: "1269152c209c4e00920666da1344783f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023df7f0b0f54dcab50420358937f62d" is_pre_election: true
I20260807 18:27:47.288800 29275 raft_consensus.cc:2454] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1269152c209c4e00920666da1344783f in term 0.
I20260807 18:27:47.289448 28991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f701b1d4e741808b2bd747b475f6ee" candidate_uuid: "1269152c209c4e00920666da1344783f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40137d2ca6e6489fa438cfcd710129fd" is_pre_election: true
I20260807 18:27:47.289685 29059 leader_election.cc:302] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023df7f0b0f54dcab50420358937f62d, 1269152c209c4e00920666da1344783f; no voters:
I20260807 18:27:47.289968 28991 raft_consensus.cc:2454] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1269152c209c4e00920666da1344783f in term 0.
I20260807 18:27:47.290189 29354 raft_consensus.cc:2781] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:27:47.290366 29354 raft_consensus.cc:482] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:27:47.290496 29354 raft_consensus.cc:3037] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:47.292949 29354 raft_consensus.cc:504] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } }
I20260807 18:27:47.293576 29354 leader_election.cc:288] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [CANDIDATE]: Term 1 election: Requested vote from peers 023df7f0b0f54dcab50420358937f62d (127.27.1.195:39883), 40137d2ca6e6489fa438cfcd710129fd (127.27.1.193:33365)
I20260807 18:27:47.294037 28991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f701b1d4e741808b2bd747b475f6ee" candidate_uuid: "1269152c209c4e00920666da1344783f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40137d2ca6e6489fa438cfcd710129fd"
I20260807 18:27:47.294106 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f701b1d4e741808b2bd747b475f6ee" candidate_uuid: "1269152c209c4e00920666da1344783f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023df7f0b0f54dcab50420358937f62d"
I20260807 18:27:47.294250 28991 raft_consensus.cc:3037] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:47.294360 29275 raft_consensus.cc:3037] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:47.296916 28991 raft_consensus.cc:2454] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1269152c209c4e00920666da1344783f in term 1.
I20260807 18:27:47.297370 29062 leader_election.cc:302] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1269152c209c4e00920666da1344783f, 40137d2ca6e6489fa438cfcd710129fd; no voters:
I20260807 18:27:47.297683 29354 raft_consensus.cc:2781] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:47.298359 29354 raft_consensus.cc:686] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 1 LEADER]: Becoming Leader. State: Replica: 1269152c209c4e00920666da1344783f, State: Running, Role: LEADER
I20260807 18:27:47.298418 29275 raft_consensus.cc:2454] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1269152c209c4e00920666da1344783f in term 1.
I20260807 18:27:47.298868 29354 consensus_queue.cc:237] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } }
I20260807 18:27:47.304236 28823 catalog_manager.cc:5495] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1269152c209c4e00920666da1344783f (127.27.1.194). New cstate: current_term: 1 leader_uuid: "1269152c209c4e00920666da1344783f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } health_report { overall_health: UNKNOWN } } }
I20260807 18:27:47.391716 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.386023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.27.1.194:34945)
0807 18:27:47.386302 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.386309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.386328 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:27:47.386664 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.386669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.386749 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.386901 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:47.387334 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.388071 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.389403 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.389805 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.389947 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.390518 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:47.391055 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:47.391253 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.391352 (+ 99us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.391435 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:27:47.399854 29275 raft_consensus.cc:1261] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Refusing update from remote peer 1269152c209c4e00920666da1344783f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:47.399854 28991 raft_consensus.cc:1261] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd [term 1 FOLLOWER]: Refusing update from remote peer 1269152c209c4e00920666da1344783f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:47.400863 29357 consensus_queue.cc:1035] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [LEADER]: Connected to new peer: Peer: permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:47.401350 29354 consensus_queue.cc:1035] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [LEADER]: Connected to new peer: Peer: permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:47.413371 29360 mvcc.cc:205] Tried to move back new op lower bound from 7315977287259922432 to 7315977286861238272. Current Snapshot: MvccSnapshot[applied={T|T < 7315977287259922432}]
I20260807 18:27:47.437736 28823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:47.438133 28823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:47.442670 28823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7: Sending DeleteTablet for 3 replicas of tablet 19f701b1d4e741808b2bd747b475f6ee
I20260807 18:27:47.443970 29254 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f701b1d4e741808b2bd747b475f6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:47 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:35242
I20260807 18:27:47.444033 29112 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f701b1d4e741808b2bd747b475f6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:47 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:35408
I20260807 18:27:47.444391 28965 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f701b1d4e741808b2bd747b475f6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:47 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:47730
I20260807 18:27:47.448102 28823 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:27:47.448904 28823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:27:47.455276 29374 tablet_replica.cc:330] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: stopping tablet replica
I20260807 18:27:47.456557 29374 raft_consensus.cc:2229] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:47.457710 29374 raft_consensus.cc:2258] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:47.460929 29374 ts_tablet_manager.cc:1905] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:47.461462 29253 tablet_service.cc:1467] Processing CreateTablet for tablet 8c32316697c54c619f7a694aee0025ca (DEFAULT_TABLE table=test-table [id=fd1ff25972ab43d4b9d93b09cc10f586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:47.461999 29111 tablet_service.cc:1467] Processing CreateTablet for tablet 8c32316697c54c619f7a694aee0025ca (DEFAULT_TABLE table=test-table [id=fd1ff25972ab43d4b9d93b09cc10f586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:47.462173 29253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c32316697c54c619f7a694aee0025ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:47.462706 29111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c32316697c54c619f7a694aee0025ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:47.467482 28965 tablet_service.cc:1467] Processing CreateTablet for tablet 8c32316697c54c619f7a694aee0025ca (DEFAULT_TABLE table=test-table [id=fd1ff25972ab43d4b9d93b09cc10f586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:47.468041 28965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c32316697c54c619f7a694aee0025ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:47.468847 29373 tablet_replica.cc:330] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: stopping tablet replica
I20260807 18:27:47.469293 29373 raft_consensus.cc:2229] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:27:47.470080 29347 tablet_bootstrap.cc:492] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d: Bootstrap starting.
I20260807 18:27:47.471411 29373 raft_consensus.cc:2258] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:47.471796 29347 tablet_bootstrap.cc:654] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:47.474680 29373 ts_tablet_manager.cc:1905] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:47.474862 29348 tablet_bootstrap.cc:492] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f: Bootstrap starting.
I20260807 18:27:47.477228 29348 tablet_bootstrap.cc:654] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:47.483942 29347 tablet_bootstrap.cc:492] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d: No bootstrap required, opened a new log
I20260807 18:27:47.484130 29346 tablet_bootstrap.cc:492] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd: Bootstrap starting.
I20260807 18:27:47.484261 29347 ts_tablet_manager.cc:1397] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260807 18:27:47.485582 29374 ts_tablet_manager.cc:1918] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:27:47.485607 29347 raft_consensus.cc:348] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.485947 29346 tablet_bootstrap.cc:654] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:47.486163 29374 log.cc:1196] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/19f701b1d4e741808b2bd747b475f6ee
I20260807 18:27:47.486215 29347 raft_consensus.cc:374] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:47.486501 29347 raft_consensus.cc:729] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023df7f0b0f54dcab50420358937f62d, State: Initialized, Role: FOLLOWER
I20260807 18:27:47.486868 29374 ts_tablet_manager.cc:1939] T 19f701b1d4e741808b2bd747b475f6ee P 40137d2ca6e6489fa438cfcd710129fd: Deleting consensus metadata
I20260807 18:27:47.486840 29347 consensus_queue.cc:260] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.487661 29347 ts_tablet_manager.cc:1428] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 18:27:47.488052 29372 tablet_replica.cc:330] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: stopping tablet replica
I20260807 18:27:47.488655 29372 raft_consensus.cc:2229] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:47.488886 28811 catalog_manager.cc:4841] TS 40137d2ca6e6489fa438cfcd710129fd (127.27.1.193:33365): tablet 19f701b1d4e741808b2bd747b475f6ee (table test-table [id=65e5d4339c26480ea68bffec3d00c3c9]) successfully deleted
I20260807 18:27:47.489249 29372 raft_consensus.cc:2258] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:47.491876 29373 ts_tablet_manager.cc:1918] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:27:47.492067 29373 log.cc:1196] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/19f701b1d4e741808b2bd747b475f6ee
I20260807 18:27:47.492309 29372 ts_tablet_manager.cc:1905] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:47.492815 29373 ts_tablet_manager.cc:1939] T 19f701b1d4e741808b2bd747b475f6ee P 1269152c209c4e00920666da1344783f: Deleting consensus metadata
I20260807 18:27:47.494776 28811 catalog_manager.cc:4841] TS 1269152c209c4e00920666da1344783f (127.27.1.194:34945): tablet 19f701b1d4e741808b2bd747b475f6ee (table test-table [id=65e5d4339c26480ea68bffec3d00c3c9]) successfully deleted
I20260807 18:27:47.498667 29372 ts_tablet_manager.cc:1918] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:27:47.498835 29372 log.cc:1196] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/19f701b1d4e741808b2bd747b475f6ee
I20260807 18:27:47.499403 29372 ts_tablet_manager.cc:1939] T 19f701b1d4e741808b2bd747b475f6ee P 023df7f0b0f54dcab50420358937f62d: Deleting consensus metadata
I20260807 18:27:47.500727 29346 tablet_bootstrap.cc:492] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd: No bootstrap required, opened a new log
I20260807 18:27:47.500933 29346 ts_tablet_manager.cc:1397] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20260807 18:27:47.501037 28808 catalog_manager.cc:4841] TS 023df7f0b0f54dcab50420358937f62d (127.27.1.195:39883): tablet 19f701b1d4e741808b2bd747b475f6ee (table test-table [id=65e5d4339c26480ea68bffec3d00c3c9]) successfully deleted
I20260807 18:27:47.501781 29346 raft_consensus.cc:348] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.502043 29346 raft_consensus.cc:374] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:47.502161 29346 raft_consensus.cc:729] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40137d2ca6e6489fa438cfcd710129fd, State: Initialized, Role: FOLLOWER
I20260807 18:27:47.502302 29348 tablet_bootstrap.cc:492] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f: No bootstrap required, opened a new log
I20260807 18:27:47.502470 29348 ts_tablet_manager.cc:1397] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.010s
I20260807 18:27:47.502445 29346 consensus_queue.cc:260] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.503330 29348 raft_consensus.cc:348] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.503567 29348 raft_consensus.cc:374] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:47.503996 29348 raft_consensus.cc:729] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1269152c209c4e00920666da1344783f, State: Initialized, Role: FOLLOWER
I20260807 18:27:47.504315 29348 consensus_queue.cc:260] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.505121 29348 ts_tablet_manager.cc:1428] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 18:27:47.504987 29346 ts_tablet_manager.cc:1428] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260807 18:27:47.581844 29352 raft_consensus.cc:482] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:47.582105 29352 raft_consensus.cc:504] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.583882 29352 leader_election.cc:288] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1269152c209c4e00920666da1344783f (127.27.1.194:34945), 023df7f0b0f54dcab50420358937f62d (127.27.1.195:39883)
I20260807 18:27:47.591706 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.584045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:34945 (local address 127.27.1.193:36307)
0807 18:27:47.584748 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:47.584771 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:47.584909 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:47.585712 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:47.585718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:47.585758 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:47.586223 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.586237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.587838 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.587844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.589582 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.589594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.590857 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.590864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.591041 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.591078 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 18:27:47.591206 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 18:27:47.591338 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":328,"threads_started":1}
I20260807 18:27:47.591704 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.584992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:36307 (local address 127.27.1.194:34945)
0807 18:27:47.585204 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.585209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.585230 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:27:47.585305 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.585310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.585396 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.585487 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:47.586435 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.587686 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.589753 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.590630 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.590816 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.590897 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.591241 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.591383 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:27:47.592731 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c32316697c54c619f7a694aee0025ca" candidate_uuid: "40137d2ca6e6489fa438cfcd710129fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1269152c209c4e00920666da1344783f" is_pre_election: true
I20260807 18:27:47.593284 29137 raft_consensus.cc:2454] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40137d2ca6e6489fa438cfcd710129fd in term 0.
I20260807 18:27:47.593564 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.585298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:45295 (local address 127.27.1.195:39883)
0807 18:27:47.585504 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.585510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.586228 (+ 718us) server_negotiation.cc:407] Connection header received
0807 18:27:47.586547 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.586551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.586626 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.586726 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:47.587383 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.588372 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.591019 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.592265 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.592424 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.592509 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.593040 (+ 531us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.593230 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 18:27:47.593691 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.584363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:39883 (local address 127.27.1.193:45295)
0807 18:27:47.586099 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:47.586118 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:47.586322 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:47.586741 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:47.586745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:47.586768 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:47.587181 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.587192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.588546 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.588554 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.590857 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.590873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.592730 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.592736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.592873 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.592896 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 18:27:47.593010 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 18:27:47.593190 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1594,"thread_start_us":1400,"threads_started":1}
I20260807 18:27:47.594040 28917 leader_election.cc:302] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1269152c209c4e00920666da1344783f, 40137d2ca6e6489fa438cfcd710129fd; no voters:
I20260807 18:27:47.594491 29352 raft_consensus.cc:2781] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:27:47.594493 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c32316697c54c619f7a694aee0025ca" candidate_uuid: "40137d2ca6e6489fa438cfcd710129fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023df7f0b0f54dcab50420358937f62d" is_pre_election: true
I20260807 18:27:47.594630 29352 raft_consensus.cc:482] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:27:47.594720 29352 raft_consensus.cc:3037] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:47.594751 29275 raft_consensus.cc:2454] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40137d2ca6e6489fa438cfcd710129fd in term 0.
I20260807 18:27:47.597477 29352 raft_consensus.cc:504] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.598091 29352 leader_election.cc:288] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [CANDIDATE]: Term 1 election: Requested vote from peers 1269152c209c4e00920666da1344783f (127.27.1.194:34945), 023df7f0b0f54dcab50420358937f62d (127.27.1.195:39883)
I20260807 18:27:47.598582 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c32316697c54c619f7a694aee0025ca" candidate_uuid: "40137d2ca6e6489fa438cfcd710129fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023df7f0b0f54dcab50420358937f62d"
I20260807 18:27:47.598583 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c32316697c54c619f7a694aee0025ca" candidate_uuid: "40137d2ca6e6489fa438cfcd710129fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1269152c209c4e00920666da1344783f"
I20260807 18:27:47.598793 29275 raft_consensus.cc:3037] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:47.598800 29137 raft_consensus.cc:3037] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:47.601938 29275 raft_consensus.cc:2454] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40137d2ca6e6489fa438cfcd710129fd in term 1.
I20260807 18:27:47.602316 29137 raft_consensus.cc:2454] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40137d2ca6e6489fa438cfcd710129fd in term 1.
I20260807 18:27:47.602424 28914 leader_election.cc:302] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023df7f0b0f54dcab50420358937f62d, 40137d2ca6e6489fa438cfcd710129fd; no voters:
I20260807 18:27:47.602761 29352 raft_consensus.cc:2781] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:47.610225 29352 raft_consensus.cc:686] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 1 LEADER]: Becoming Leader. State: Replica: 40137d2ca6e6489fa438cfcd710129fd, State: Running, Role: LEADER
I20260807 18:27:47.610643 29352 consensus_queue.cc:237] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.614553 28823 catalog_manager.cc:5495] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 40137d2ca6e6489fa438cfcd710129fd (127.27.1.193). New cstate: current_term: 1 leader_uuid: "40137d2ca6e6489fa438cfcd710129fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } health_report { overall_health: UNKNOWN } } }
Aug 07 18:27:47 dist-test-slave-1xrh krb5kdc[28778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127265, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:47.683404 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.661576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42065 (local address 127.0.0.1:40244)
0807 18:27:47.662510 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:47.662527 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:47.662651 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:47.663009 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:47.663012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:47.663852 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:47.664067 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.664077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.665623 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.665627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.666596 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.666607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.667668 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.667673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.667796 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.674634 (+ 6838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:47.674658 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:47.677348 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:47.680497 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:47.680510 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:47.680518 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:47.681060 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:47.681399 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:47.681403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:47.681407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:47.681687 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:47.682253 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:47.682262 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:47.682447 (+ 185us) client_negotiation.cc:769] Sending connection context
0807 18:27:47.682616 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 18:27:47.682917 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":639,"threads_started":1}
I20260807 18:27:47.683983 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.661577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.27.1.254:42065)
0807 18:27:47.662226 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.662232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.662613 (+ 381us) server_negotiation.cc:407] Connection header received
0807 18:27:47.662761 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.662768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.662842 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.662972 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:47.664246 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.665490 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.666745 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.667495 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.667638 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.677581 (+ 9943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:47.677586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:47.677590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:47.677621 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:47.680363 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:47.681171 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:47.681175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:47.681177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:47.681294 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:47.681793 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:47.681796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:47.681798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:47.682122 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:47.682234 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.682691 (+ 457us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.683041 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":171,"server-negotiator.queue_time_us":503,"thread_start_us":337,"threads_started":1}
I20260807 18:27:47.689244 28823 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:40244:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 18:27:47.690538 28823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:27:47.698695 29254 tablet_service.cc:1467] Processing CreateTablet for tablet a7842247a46a41c0afb7033680227ce9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d0deeea4d594359aa30c21ad06a11d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 18:27:47.698590 28965 tablet_service.cc:1467] Processing CreateTablet for tablet a7842247a46a41c0afb7033680227ce9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d0deeea4d594359aa30c21ad06a11d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 18:27:47.699324 29254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7842247a46a41c0afb7033680227ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:47.699409 28965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7842247a46a41c0afb7033680227ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:47.705082 29111 tablet_service.cc:1467] Processing CreateTablet for tablet a7842247a46a41c0afb7033680227ce9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d0deeea4d594359aa30c21ad06a11d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 18:27:47.705300 29347 tablet_bootstrap.cc:492] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: Bootstrap starting.
I20260807 18:27:47.705690 29111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7842247a46a41c0afb7033680227ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:47.706705 29347 tablet_bootstrap.cc:654] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:47.709059 29347 tablet_bootstrap.cc:492] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: No bootstrap required, opened a new log
I20260807 18:27:47.709319 29347 ts_tablet_manager.cc:1397] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 18:27:47.709937 29347 raft_consensus.cc:348] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.710157 29347 raft_consensus.cc:374] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:47.710263 29347 raft_consensus.cc:729] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023df7f0b0f54dcab50420358937f62d, State: Initialized, Role: FOLLOWER
I20260807 18:27:47.710536 29347 consensus_queue.cc:260] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.711074 29353 tablet_replica.cc:439] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } }
I20260807 18:27:47.711480 29353 tablet_replica.cc:442] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:47.711937 29347 ts_tablet_manager.cc:1428] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 18:27:47.713696 29348 tablet_bootstrap.cc:492] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: Bootstrap starting.
I20260807 18:27:47.714625 29346 tablet_bootstrap.cc:492] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: Bootstrap starting.
I20260807 18:27:47.715610 29348 tablet_bootstrap.cc:654] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:47.716471 29346 tablet_bootstrap.cc:654] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:47.719305 29346 tablet_bootstrap.cc:492] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: No bootstrap required, opened a new log
I20260807 18:27:47.719537 29346 ts_tablet_manager.cc:1397] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 18:27:47.720132 29346 raft_consensus.cc:348] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.720348 29346 raft_consensus.cc:374] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:47.720430 29346 raft_consensus.cc:729] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40137d2ca6e6489fa438cfcd710129fd, State: Initialized, Role: FOLLOWER
I20260807 18:27:47.720696 29346 consensus_queue.cc:260] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.721088 29352 tablet_replica.cc:439] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } }
I20260807 18:27:47.721415 29346 ts_tablet_manager.cc:1428] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260807 18:27:47.721422 29352 tablet_replica.cc:442] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:47.723428 29348 tablet_bootstrap.cc:492] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: No bootstrap required, opened a new log
I20260807 18:27:47.723631 29348 ts_tablet_manager.cc:1397] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260807 18:27:47.724455 29348 raft_consensus.cc:348] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.724746 29348 raft_consensus.cc:374] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:47.724866 29348 raft_consensus.cc:729] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1269152c209c4e00920666da1344783f, State: Initialized, Role: FOLLOWER
I20260807 18:27:47.725140 29348 consensus_queue.cc:260] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.725651 29368 tablet_replica.cc:439] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } }
I20260807 18:27:47.726109 29368 tablet_replica.cc:442] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:47.726492 29348 ts_tablet_manager.cc:1428] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 18:27:47.926213 29353 raft_consensus.cc:482] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:47.926440 29353 raft_consensus.cc:504] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.928115 29353 leader_election.cc:288] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40137d2ca6e6489fa438cfcd710129fd (127.27.1.193:33365), 1269152c209c4e00920666da1344783f (127.27.1.194:34945)
I20260807 18:27:47.929559 29352 raft_consensus.cc:482] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:47.929771 29352 raft_consensus.cc:504] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.930438 29352 leader_election.cc:288] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1269152c209c4e00920666da1344783f (127.27.1.194:34945), 023df7f0b0f54dcab50420358937f62d (127.27.1.195:39883)
I20260807 18:27:47.931187 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7842247a46a41c0afb7033680227ce9" candidate_uuid: "40137d2ca6e6489fa438cfcd710129fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023df7f0b0f54dcab50420358937f62d" is_pre_election: true
I20260807 18:27:47.931494 29275 raft_consensus.cc:2454] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40137d2ca6e6489fa438cfcd710129fd in term 0.
I20260807 18:27:47.931978 28914 leader_election.cc:302] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023df7f0b0f54dcab50420358937f62d, 40137d2ca6e6489fa438cfcd710129fd; no voters:
I20260807 18:27:47.932268 29352 raft_consensus.cc:2781] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:27:47.932354 29352 raft_consensus.cc:482] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:27:47.932444 29352 raft_consensus.cc:3037] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:47.935015 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7842247a46a41c0afb7033680227ce9" candidate_uuid: "40137d2ca6e6489fa438cfcd710129fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1269152c209c4e00920666da1344783f" is_pre_election: true
I20260807 18:27:47.935284 29137 raft_consensus.cc:2454] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40137d2ca6e6489fa438cfcd710129fd in term 0.
I20260807 18:27:47.935721 29352 raft_consensus.cc:504] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.936951 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7842247a46a41c0afb7033680227ce9" candidate_uuid: "40137d2ca6e6489fa438cfcd710129fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023df7f0b0f54dcab50420358937f62d"
I20260807 18:27:47.937179 29275 raft_consensus.cc:3037] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:47.938234 29352 leader_election.cc:288] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [CANDIDATE]: Term 1 election: Requested vote from peers 1269152c209c4e00920666da1344783f (127.27.1.194:34945), 023df7f0b0f54dcab50420358937f62d (127.27.1.195:39883)
I20260807 18:27:47.938807 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7842247a46a41c0afb7033680227ce9" candidate_uuid: "40137d2ca6e6489fa438cfcd710129fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1269152c209c4e00920666da1344783f"
I20260807 18:27:47.939033 29137 raft_consensus.cc:3037] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:47.939270 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.929329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:33191 (local address 127.27.1.194:34945)
0807 18:27:47.929516 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.929523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.930150 (+ 627us) server_negotiation.cc:407] Connection header received
0807 18:27:47.930408 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.930414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.930499 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.930571 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:47.931603 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.932667 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.936731 (+ 4064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.937848 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.937989 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.938068 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.938775 (+ 707us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.938927 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 18:27:47.940387 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.929221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:34945 (local address 127.27.1.195:33191)
0807 18:27:47.930052 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:47.930072 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:47.930243 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:47.930835 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:47.930840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:47.930893 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:47.931327 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.931341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.932942 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.932946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.936534 (+ 3588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.936548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.938238 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.938243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.938587 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.938622 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 18:27:47.938748 (+ 126us) client_negotiation.cc:240] Negotiation successful
0807 18:27:47.939034 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":3975,"threads_started":1}
I20260807 18:27:47.941280 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7842247a46a41c0afb7033680227ce9" candidate_uuid: "023df7f0b0f54dcab50420358937f62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1269152c209c4e00920666da1344783f" is_pre_election: true
I20260807 18:27:47.942155 29137 raft_consensus.cc:2454] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40137d2ca6e6489fa438cfcd710129fd in term 1.
I20260807 18:27:47.942574 28917 leader_election.cc:302] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1269152c209c4e00920666da1344783f, 40137d2ca6e6489fa438cfcd710129fd; no voters:
I20260807 18:27:47.942873 29352 raft_consensus.cc:2781] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:47.943116 29352 raft_consensus.cc:686] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [term 1 LEADER]: Becoming Leader. State: Replica: 40137d2ca6e6489fa438cfcd710129fd, State: Running, Role: LEADER
I20260807 18:27:47.943302 29275 raft_consensus.cc:2454] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40137d2ca6e6489fa438cfcd710129fd in term 1.
I20260807 18:27:47.943558 29352 consensus_queue.cc:237] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } }
I20260807 18:27:47.945094 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.928699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:46713 (local address 127.27.1.193:33365)
0807 18:27:47.934710 (+ 6011us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:47.934717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:47.934735 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:27:47.934790 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:47.934796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:47.934863 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:47.935116 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:47.936513 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.937817 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.942239 (+ 4422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.943950 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.944175 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.944276 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:47.944614 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 18:27:47.944757 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5844,"thread_start_us":5720,"threads_started":1}
I20260807 18:27:47.945210 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:47.928330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:33365 (local address 127.27.1.195:46713)
0807 18:27:47.928931 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:47.928952 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:47.929115 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:47.935025 (+ 5910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:47.935031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:47.935056 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:47.936283 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.936298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.937990 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.937995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.941696 (+ 3701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:47.941711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:47.944244 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:47.944252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:47.944422 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:47.944466 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 18:27:47.944570 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 18:27:47.944757 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":482,"threads_started":1}
I20260807 18:27:47.945883 28991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7842247a46a41c0afb7033680227ce9" candidate_uuid: "023df7f0b0f54dcab50420358937f62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40137d2ca6e6489fa438cfcd710129fd" is_pre_election: true
I20260807 18:27:47.946718 29207 leader_election.cc:302] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 023df7f0b0f54dcab50420358937f62d; no voters: 1269152c209c4e00920666da1344783f, 40137d2ca6e6489fa438cfcd710129fd
I20260807 18:27:47.947630 29353 raft_consensus.cc:2726] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 18:27:47.952020 29380 tablet_replica.cc:439] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: TxnStatusTablet state changed. Reason: New leader 40137d2ca6e6489fa438cfcd710129fd. Latest consensus state: current_term: 1 leader_uuid: "40137d2ca6e6489fa438cfcd710129fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } }
I20260807 18:27:47.952435 29380 tablet_replica.cc:442] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: This TxnStatusTablet replica's current role is: LEADER
I20260807 18:27:47.953223 28826 catalog_manager.cc:5495] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 40137d2ca6e6489fa438cfcd710129fd (127.27.1.193). New cstate: current_term: 1 leader_uuid: "40137d2ca6e6489fa438cfcd710129fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } health_report { overall_health: UNKNOWN } } }
I20260807 18:27:47.954741 29396 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 18:27:47.955025 29396 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 18:27:48.019059 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:48.012873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.27.1.193:33365)
0807 18:27:48.013038 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:48.013042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:48.013061 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:27:48.013345 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:48.013349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:48.013425 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:48.013548 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:48.013964 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.014767 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.016760 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.017203 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.017303 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:48.017895 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:48.018459 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:48.018584 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:48.018717 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 18:27:48.018788 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 18:27:48.019222 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:48.012710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:33365 (local address 127.0.0.1:47740)
0807 18:27:48.012981 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:48.013005 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:48.013145 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:48.013596 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:48.013602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:48.013616 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 18:27:48.013814 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:48.013830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.014908 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.014914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:48.016601 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:48.016618 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.017354 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.017360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:48.017577 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:48.017660 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 18:27:48.018574 (+ 914us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 18:27:48.018595 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 18:27:48.018686 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 18:27:48.018851 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 18:27:48.024236 29275 raft_consensus.cc:1261] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Refusing update from remote peer 40137d2ca6e6489fa438cfcd710129fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:48.024307 29137 raft_consensus.cc:1261] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f [term 1 FOLLOWER]: Refusing update from remote peer 40137d2ca6e6489fa438cfcd710129fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:48.025000 29380 consensus_queue.cc:1035] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:48.025354 29352 consensus_queue.cc:1035] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:48.032665 29368 tablet_replica.cc:439] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: TxnStatusTablet state changed. Reason: New leader 40137d2ca6e6489fa438cfcd710129fd. Latest consensus state: current_term: 1 leader_uuid: "40137d2ca6e6489fa438cfcd710129fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } }
I20260807 18:27:48.033124 29368 tablet_replica.cc:442] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:48.036053 29380 tablet_replica.cc:439] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40137d2ca6e6489fa438cfcd710129fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } }
I20260807 18:27:48.036377 29380 tablet_replica.cc:442] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: This TxnStatusTablet replica's current role is: LEADER
I20260807 18:27:48.038898 29399 mvcc.cc:205] Tried to move back new op lower bound from 7315977289819725824 to 7315977289499979776. Current Snapshot: MvccSnapshot[applied={T|T < 7315977289819725824 or (T in {7315977289819725824})}]
I20260807 18:27:48.042860 29368 tablet_replica.cc:439] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40137d2ca6e6489fa438cfcd710129fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } }
I20260807 18:27:48.043107 29368 tablet_replica.cc:442] T a7842247a46a41c0afb7033680227ce9 P 1269152c209c4e00920666da1344783f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:48.044849 29353 tablet_replica.cc:439] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: TxnStatusTablet state changed. Reason: New leader 40137d2ca6e6489fa438cfcd710129fd. Latest consensus state: current_term: 1 leader_uuid: "40137d2ca6e6489fa438cfcd710129fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } }
I20260807 18:27:48.045212 29353 tablet_replica.cc:442] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:48.047408 29352 tablet_replica.cc:439] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40137d2ca6e6489fa438cfcd710129fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } }
I20260807 18:27:48.047714 29352 tablet_replica.cc:442] T a7842247a46a41c0afb7033680227ce9 P 40137d2ca6e6489fa438cfcd710129fd: This TxnStatusTablet replica's current role is: LEADER
I20260807 18:27:48.050369 29400 mvcc.cc:205] Tried to move back new op lower bound from 7315977289819725824 to 7315977289499979776. Current Snapshot: MvccSnapshot[applied={T|T < 7315977289819725824}]
I20260807 18:27:48.055694 29353 tablet_replica.cc:439] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40137d2ca6e6489fa438cfcd710129fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40137d2ca6e6489fa438cfcd710129fd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 33365 } } peers { permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 } } peers { permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 } } }
I20260807 18:27:48.056011 29353 tablet_replica.cc:442] T a7842247a46a41c0afb7033680227ce9 P 023df7f0b0f54dcab50420358937f62d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:48.065764 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:48.049822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.27.1.193:33365)
0807 18:27:48.050047 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:48.050056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:48.059804 (+ 9748us) server_negotiation.cc:407] Connection header received
0807 18:27:48.060006 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:48.060010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:48.060072 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:48.060157 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:48.060757 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.061599 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.063691 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.064187 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.064352 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:48.064814 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:48.064919 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:48.064993 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:48.065207 (+ 214us) server_negotiation.cc:299] Negotiation successful
0807 18:27:48.065277 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 18:27:48.083952 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:48.077208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:47587 (local address 127.27.1.193:33365)
0807 18:27:48.077407 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:48.077411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:48.077869 (+ 458us) server_negotiation.cc:407] Connection header received
0807 18:27:48.078054 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:48.078057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:48.078116 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:48.078199 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:48.078882 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.079661 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.082063 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.082561 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.082655 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:48.083231 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:48.083360 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:48.083434 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:48.083587 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 18:27:48.083676 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":83}
I20260807 18:27:48.084287 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:48.076981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:33365 (local address 127.27.1.193:47587)
0807 18:27:48.077762 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:48.077782 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:48.077932 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:48.078247 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:48.078255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:48.078270 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 18:27:48.078701 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:48.078719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.079803 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.079809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:48.081901 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:48.081922 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.082748 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.082754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:48.082942 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:48.083029 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 18:27:48.083460 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 18:27:48.083472 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 18:27:48.083562 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 18:27:48.083726 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":4,"thread_start_us":421,"threads_started":1}
I20260807 18:27:48.095489 29137 raft_consensus.cc:1261] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f [term 1 FOLLOWER]: Refusing update from remote peer 40137d2ca6e6489fa438cfcd710129fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:48.095485 29275 raft_consensus.cc:1261] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Refusing update from remote peer 40137d2ca6e6489fa438cfcd710129fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:48.096050 29407 consensus_queue.cc:1035] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1269152c209c4e00920666da1344783f" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:48.096531 29380 consensus_queue.cc:1035] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "023df7f0b0f54dcab50420358937f62d" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 39883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:48.157394 28826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:48.157671 28826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:48.160737 28826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d1b280a013e4af1920e1a4495bedbc7: Sending DeleteTablet for 3 replicas of tablet 8c32316697c54c619f7a694aee0025ca
I20260807 18:27:48.161582 29111 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c32316697c54c619f7a694aee0025ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:48 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:35408
I20260807 18:27:48.161733 28965 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c32316697c54c619f7a694aee0025ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:48 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:47730
I20260807 18:27:48.162026 29254 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c32316697c54c619f7a694aee0025ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:48 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:35242
I20260807 18:27:48.162352 29416 tablet_replica.cc:330] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f: stopping tablet replica
I20260807 18:27:48.162551 29416 raft_consensus.cc:2229] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:48.162803 29416 raft_consensus.cc:2258] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:48.168394 29416 ts_tablet_manager.cc:1905] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:48.175201 29417 tablet_replica.cc:330] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd: stopping tablet replica
I20260807 18:27:48.175509 29417 raft_consensus.cc:2229] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:27:48.176246 29417 raft_consensus.cc:2258] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:48.177495 29416 ts_tablet_manager.cc:1918] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:27:48.177713 29417 ts_tablet_manager.cc:1905] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:48.178326 29416 log.cc:1196] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/8c32316697c54c619f7a694aee0025ca
I20260807 18:27:48.179032 29416 ts_tablet_manager.cc:1939] T 8c32316697c54c619f7a694aee0025ca P 1269152c209c4e00920666da1344783f: Deleting consensus metadata
I20260807 18:27:48.180812 28811 catalog_manager.cc:4841] TS 1269152c209c4e00920666da1344783f (127.27.1.194:34945): tablet 8c32316697c54c619f7a694aee0025ca (table test-table [id=fd1ff25972ab43d4b9d93b09cc10f586]) successfully deleted
I20260807 18:27:48.184918 29418 tablet_replica.cc:330] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d: stopping tablet replica
I20260807 18:27:48.185195 29418 raft_consensus.cc:2229] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:48.185820 29418 raft_consensus.cc:2258] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:48.185879 29417 ts_tablet_manager.cc:1918] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:27:48.186043 29417 log.cc:1196] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/8c32316697c54c619f7a694aee0025ca
I20260807 18:27:48.187151 29418 ts_tablet_manager.cc:1905] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:48.187502 29417 ts_tablet_manager.cc:1939] T 8c32316697c54c619f7a694aee0025ca P 40137d2ca6e6489fa438cfcd710129fd: Deleting consensus metadata
I20260807 18:27:48.188649 28811 catalog_manager.cc:4841] TS 40137d2ca6e6489fa438cfcd710129fd (127.27.1.193:33365): tablet 8c32316697c54c619f7a694aee0025ca (table test-table [id=fd1ff25972ab43d4b9d93b09cc10f586]) successfully deleted
Aug 07 18:27:48 dist-test-slave-1xrh krb5kdc[28778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127266, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:27:48.194906 29418 ts_tablet_manager.cc:1918] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:27:48.195045 29418 log.cc:1196] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/8c32316697c54c619f7a694aee0025ca
I20260807 18:27:48.195700 29418 ts_tablet_manager.cc:1939] T 8c32316697c54c619f7a694aee0025ca P 023df7f0b0f54dcab50420358937f62d: Deleting consensus metadata
I20260807 18:27:48.196939 28808 catalog_manager.cc:4841] TS 023df7f0b0f54dcab50420358937f62d (127.27.1.195:39883): tablet 8c32316697c54c619f7a694aee0025ca (table test-table [id=fd1ff25972ab43d4b9d93b09cc10f586]) successfully deleted
I20260807 18:27:48.198354 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:48.164360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.27.1.193:33365)
0807 18:27:48.164591 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:48.164600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:48.179992 (+ 15392us) server_negotiation.cc:407] Connection header received
0807 18:27:48.180199 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:48.180206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:48.180288 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:48.180410 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:48.182685 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.184026 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.186039 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.186701 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.186847 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:48.191603 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:48.191628 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:48.191663 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:48.191698 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:48.194934 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:48.195869 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:48.195878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:48.195882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:48.196045 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:48.196530 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:48.196536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:48.196539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:48.197108 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:48.197565 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:48.197730 (+ 165us) server_negotiation.cc:299] Negotiation successful
0807 18:27:48.197974 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 18:27:48.199203 29034 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47770
I20260807 18:27:48.220484 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:48.202411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.27.1.254:42065)
0807 18:27:48.203018 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:48.203024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:48.203596 (+ 572us) server_negotiation.cc:407] Connection header received
0807 18:27:48.203739 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:48.203746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:48.203842 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:48.203965 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:48.205586 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.207003 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.208829 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.209522 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.209616 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:48.212900 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:48.212908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:48.212913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:48.212981 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:48.217399 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:48.218203 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:48.218210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:48.218214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:48.218414 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:48.218825 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:48.218831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:48.218835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:48.219210 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:48.219467 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:48.219912 (+ 445us) server_negotiation.cc:299] Negotiation successful
0807 18:27:48.220131 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":357,"threads_started":1}
W20260807 18:27:48.221050 28826 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40254
I20260807 18:27:48.223568 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 28898
I20260807 18:27:48.241149 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:48.225190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.27.1.193:33365)
0807 18:27:48.225395 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:48.225401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:48.227653 (+ 2252us) server_negotiation.cc:407] Connection header received
0807 18:27:48.227825 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:48.227831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:48.227907 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:48.228007 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:48.229672 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.230413 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.232076 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.232490 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.232578 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:48.235352 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:48.235357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:48.235379 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:48.235415 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:48.238455 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:48.239271 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:48.239275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:48.239277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:48.239419 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:48.239787 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:48.239790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:48.239792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:48.240087 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:48.240260 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:48.240639 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 18:27:48.240802 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 18:27:48.241818 29034 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47772
W20260807 18:27:48.242446 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 28898: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:27:48.242632 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 28898
I20260807 18:27:48.265978 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29043
Aug 07 18:27:48 dist-test-slave-1xrh krb5kdc[28778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127266, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:27:48.281368 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:48.266724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.27.1.194:34945)
0807 18:27:48.266918 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:48.266927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:48.266949 (+ 22us) server_negotiation.cc:407] Connection header received
0807 18:27:48.267194 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:48.267200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:48.267307 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:48.267424 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:48.268621 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.269353 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.271109 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.271642 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.271742 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:48.275873 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:48.275886 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:48.275901 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:48.275934 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:48.278599 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:48.279316 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:48.279324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:48.279326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:48.279485 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:48.279871 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:48.279874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:48.279876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:48.280204 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:48.280449 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:48.280815 (+ 366us) server_negotiation.cc:299] Negotiation successful
0807 18:27:48.280992 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260807 18:27:48.282069 29178 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35424
W20260807 18:27:48.282689 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29043: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:27:48.282855 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 29043
I20260807 18:27:48.308905 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29187
Aug 07 18:27:48 dist-test-slave-1xrh krb5kdc[28778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127266, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:27:48.323936 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:48.309608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.27.1.195:39883)
0807 18:27:48.310120 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:48.310128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:48.310155 (+ 27us) server_negotiation.cc:407] Connection header received
0807 18:27:48.310210 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:48.310213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:48.310273 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:48.310394 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:48.311738 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.312422 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.313678 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:48.314125 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:48.314270 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:48.318375 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:48.318390 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:48.318408 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:48.318449 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:48.321453 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:48.322059 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:48.322066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:48.322069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:48.322187 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:48.322490 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:48.322493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:48.322495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:48.322834 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:48.323087 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:48.323353 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 18:27:48.323541 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":252,"threads_started":1}
W20260807 18:27:48.324604 29321 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35246
W20260807 18:27:48.325187 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29187: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:27:48.325315 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 29187
I20260807 18:27:48.347407 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 28791
I20260807 18:27:48.348492 28867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:27:48.466250 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 28791
2026-08-07T18:27:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5012 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:27:48 dist-test-slave-1xrh krb5kdc[29430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:27:48 dist-test-slave-1xrh krb5kdc[29430](info): set up 2 sockets
Aug 07 18:27:48 dist-test-slave-1xrh krb5kdc[29430](info): commencing operation
krb5kdc: starting...
W20260807 18:27:49.605430 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:27:49 dist-test-slave-1xrh krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:27:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:27:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:27:49.811932 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44077
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:44077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:50.140797 29444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:50.141083 29444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:50.141249 29444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:50.148326 29444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:27:50.148422 29444 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 18:27:50.148514 29444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:50.148609 29444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:50.148691 29444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:27:50.148782 29444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:27:50.158991 29444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:44077
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:50.160310 29444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:50.161670 29444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:50.173812 29449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:50.173863 29450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:50.173808 29452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:50.174842 29444 server_base.cc:1034] running on GCE node
I20260807 18:27:50.175676 29444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:50.177073 29444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:50.178328 29444 hybrid_clock.cc:648] HybridClock initialized: now 1786127270178251 us; error 51 us; skew 500 ppm
I20260807 18:27:50.178656 29444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:50 dist-test-slave-1xrh krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:50.182062 29444 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:27:50.183545 29444 webserver.cc:460] Webserver started at http://127.27.1.254:40931/ using document root <none> and password file <none>
I20260807 18:27:50.184278 29444 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:50.184419 29444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:50.184774 29444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:50.188148 29444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "e188b524b9b7470baf52cdbb1f0feca1"
format_stamp: "Formatted at 2026-08-07 18:27:50 on dist-test-slave-1xrh"
server_key: "5afd64d2ee7df6c0f67a75a580e077bb"
server_key_iv: "f67021660b42b76f365eaba33e5e8305"
server_key_version: "encryptionkey@0"
I20260807 18:27:50.188971 29444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "e188b524b9b7470baf52cdbb1f0feca1"
format_stamp: "Formatted at 2026-08-07 18:27:50 on dist-test-slave-1xrh"
server_key: "5afd64d2ee7df6c0f67a75a580e077bb"
server_key_iv: "f67021660b42b76f365eaba33e5e8305"
server_key_version: "encryptionkey@0"
I20260807 18:27:50.194231 29444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:27:50.197652 29461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:50.198936 29444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 18:27:50.199155 29444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "e188b524b9b7470baf52cdbb1f0feca1"
format_stamp: "Formatted at 2026-08-07 18:27:50 on dist-test-slave-1xrh"
server_key: "5afd64d2ee7df6c0f67a75a580e077bb"
server_key_iv: "f67021660b42b76f365eaba33e5e8305"
server_key_version: "encryptionkey@0"
I20260807 18:27:50.199393 29444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:50.218581 29444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:50.223264 29444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:50.223559 29444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:50.245009 29444 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:44077
I20260807 18:27:50.245018 29524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:44077 every 8 connection(s)
I20260807 18:27:50.246381 29444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:27:50.250324 29525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:50.250905 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 29444
I20260807 18:27:50.251113 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:27:50.251799 27655 external_mini_cluster.cc:1428] Setting key 70d74ef8c457dceadc505f8faaca5d91
I20260807 18:27:50.264837 29525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1: Bootstrap starting.
Aug 07 18:27:50 dist-test-slave-1xrh krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:50.267987 29525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:50.269482 29525 log.cc:824] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:50.273044 29525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1: No bootstrap required, opened a new log
I20260807 18:27:50.276623 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:50.254030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.27.1.254:44077)
0807 18:27:50.254890 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:50.254904 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:50.254958 (+ 54us) server_negotiation.cc:407] Connection header received
0807 18:27:50.257094 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:50.257121 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:50.257630 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:50.258054 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:50.259397 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:50.261444 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:50.262715 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:50.263746 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:50.263983 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:50.267975 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:50.267987 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:50.268007 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:50.268061 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:50.272046 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:50.272676 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:50.272684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:50.272691 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:50.272907 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:50.273287 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:50.273301 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:50.273307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:50.274035 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:50.274318 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:50.274781 (+ 463us) server_negotiation.cc:299] Negotiation successful
0807 18:27:50.275269 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":362,"threads_started":1}
I20260807 18:27:50.282346 29525 raft_consensus.cc:348] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e188b524b9b7470baf52cdbb1f0feca1" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44077 } }
I20260807 18:27:50.282683 29525 raft_consensus.cc:374] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:50.282754 29525 raft_consensus.cc:729] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e188b524b9b7470baf52cdbb1f0feca1, State: Initialized, Role: FOLLOWER
I20260807 18:27:50.283314 29525 consensus_queue.cc:260] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e188b524b9b7470baf52cdbb1f0feca1" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44077 } }
I20260807 18:27:50.283618 29525 raft_consensus.cc:388] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:27:50.283774 29525 raft_consensus.cc:482] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:27:50.283977 29525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:50.286073 29525 raft_consensus.cc:504] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e188b524b9b7470baf52cdbb1f0feca1" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44077 } }
I20260807 18:27:50.286545 29525 leader_election.cc:302] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e188b524b9b7470baf52cdbb1f0feca1; no voters:
I20260807 18:27:50.287088 29525 leader_election.cc:288] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:27:50.287277 29530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:50.288127 29530 raft_consensus.cc:686] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [term 1 LEADER]: Becoming Leader. State: Replica: e188b524b9b7470baf52cdbb1f0feca1, State: Running, Role: LEADER
I20260807 18:27:50.288650 29530 consensus_queue.cc:237] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e188b524b9b7470baf52cdbb1f0feca1" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44077 } }
I20260807 18:27:50.289322 29525 sys_catalog.cc:564] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:27:50.292040 29532 sys_catalog.cc:455] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e188b524b9b7470baf52cdbb1f0feca1. Latest consensus state: current_term: 1 leader_uuid: "e188b524b9b7470baf52cdbb1f0feca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e188b524b9b7470baf52cdbb1f0feca1" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44077 } } }
I20260807 18:27:50.292115 29531 sys_catalog.cc:455] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e188b524b9b7470baf52cdbb1f0feca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e188b524b9b7470baf52cdbb1f0feca1" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44077 } } }
I20260807 18:27:50.292428 29532 sys_catalog.cc:458] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [sys.catalog]: This master's current role is: LEADER
I20260807 18:27:50.292475 29531 sys_catalog.cc:458] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1 [sys.catalog]: This master's current role is: LEADER
I20260807 18:27:50.294728 29537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:27:50.300271 29537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:27:50.308740 29537 catalog_manager.cc:1347] Generated new cluster ID: ec69382de66d461da52a3d9be2a427c1
I20260807 18:27:50.308851 29537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:27:50.346093 29537 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:27:50.347584 29537 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:27:50.359445 29537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1: Generated new TSK 0
I20260807 18:27:50.360337 29537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:27:50.419054 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44077
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:50.753978 29551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:50.754240 29551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:50.754340 29551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:50.762428 29551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:50.762537 29551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:50.762624 29551 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 18:27:50.762677 29551 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 18:27:50.762746 29551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:27:50.773061 29551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.1.254:44077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:50.774371 29551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:50.776229 29551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:50.791445 29558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:50.791487 29557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:50.791612 29560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:50.792744 29551 server_base.cc:1034] running on GCE node
I20260807 18:27:50.793373 29551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:50.794509 29551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:50.795706 29551 hybrid_clock.cc:648] HybridClock initialized: now 1786127270795651 us; error 50 us; skew 500 ppm
I20260807 18:27:50.796000 29551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:50 dist-test-slave-1xrh krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:50.799314 29551 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:27:50.800729 29551 webserver.cc:460] Webserver started at http://127.27.1.193:38547/ using document root <none> and password file <none>
I20260807 18:27:50.801425 29551 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:50.801566 29551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:50.801904 29551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:50.805294 29551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "b368a5ecd24349eeb7e48c438d852b96"
format_stamp: "Formatted at 2026-08-07 18:27:50 on dist-test-slave-1xrh"
server_key: "439758319e62b88b9244feb6f2a51836"
server_key_iv: "eb059c1625766bd89a511cbc4edc43ed"
server_key_version: "encryptionkey@0"
I20260807 18:27:50.806169 29551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "b368a5ecd24349eeb7e48c438d852b96"
format_stamp: "Formatted at 2026-08-07 18:27:50 on dist-test-slave-1xrh"
server_key: "439758319e62b88b9244feb6f2a51836"
server_key_iv: "eb059c1625766bd89a511cbc4edc43ed"
server_key_version: "encryptionkey@0"
I20260807 18:27:50.811623 29551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 18:27:50.815761 29568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:50.817492 29551 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 18:27:50.817662 29551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "b368a5ecd24349eeb7e48c438d852b96"
format_stamp: "Formatted at 2026-08-07 18:27:50 on dist-test-slave-1xrh"
server_key: "439758319e62b88b9244feb6f2a51836"
server_key_iv: "eb059c1625766bd89a511cbc4edc43ed"
server_key_version: "encryptionkey@0"
I20260807 18:27:50.817890 29551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:50.848639 29551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:50.855020 29551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:50.855451 29551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:50.859205 29551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:50.859362 29551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:50.859489 29551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:50.859580 29551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 18:27:50 dist-test-slave-1xrh krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:50.883006 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:50.859489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:50609 (local address 127.27.1.254:44077)
0807 18:27:50.860202 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:50.860209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:50.863207 (+ 2998us) server_negotiation.cc:407] Connection header received
0807 18:27:50.864245 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:50.864251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:50.864346 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:50.864508 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:50.866703 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:50.867538 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:50.869630 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:50.870049 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:50.870191 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:50.875752 (+ 5561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:50.875762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:50.875768 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:50.875818 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:50.879521 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:50.880388 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:50.880398 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:50.880402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:50.880541 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:50.881100 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:50.881104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:50.881106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:50.881414 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:50.881587 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:50.882356 (+ 769us) server_negotiation.cc:299] Negotiation successful
0807 18:27:50.882605 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":415,"threads_started":1}
I20260807 18:27:50.884402 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:50.860297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44077 (local address 127.27.1.193:50609)
0807 18:27:50.861184 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:50.861230 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:50.863923 (+ 2693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:50.864761 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:50.864768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:50.865452 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:50.866520 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:50.866540 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:50.867717 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:50.867722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:50.869476 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:50.869490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:50.870311 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:50.870319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:50.870563 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:50.871571 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:50.871603 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:50.875445 (+ 3842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:50.879686 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:50.879696 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:50.879714 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:50.880224 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:50.880672 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:50.880677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:50.880681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:50.880894 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:50.881558 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:50.881567 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:50.881958 (+ 391us) client_negotiation.cc:769] Sending connection context
0807 18:27:50.882253 (+ 295us) client_negotiation.cc:240] Negotiation successful
0807 18:27:50.882828 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":416,"threads_started":1}
I20260807 18:27:50.907282 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:50.892325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:53667 (local address 127.27.1.254:44077)
0807 18:27:50.892542 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:50.892548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:50.893819 (+ 1271us) server_negotiation.cc:407] Connection header received
0807 18:27:50.894099 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:50.894106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:50.894224 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:50.894328 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:50.895962 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:50.896945 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:50.898280 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:50.898914 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:50.899068 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:50.901817 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:50.901823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:50.901828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:50.901869 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:50.904606 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:50.905283 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:50.905290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:50.905294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:50.905428 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:50.905808 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:50.905815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:50.905819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:50.906088 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:50.906245 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:50.906639 (+ 394us) server_negotiation.cc:299] Negotiation successful
0807 18:27:50.906874 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 18:27:50.907397 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:50.892760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44077 (local address 127.27.1.193:53667)
0807 18:27:50.893701 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:50.893726 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:50.893940 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:50.894385 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:50.894392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:50.895184 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:50.895757 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:50.895779 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:50.897100 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:50.897104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:50.898136 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:50.898147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:50.899333 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:50.899337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:50.899529 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:50.900350 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:50.900368 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:50.901609 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:50.904748 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:50.904753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:50.904757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:50.905163 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:50.905561 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:50.905566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:50.905570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:50.905697 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:50.906208 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:50.906213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:50.906398 (+ 185us) client_negotiation.cc:769] Sending connection context
0807 18:27:50.906546 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 18:27:50.906886 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":520,"threads_started":1}
I20260807 18:27:50.920315 29551 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:36365
I20260807 18:27:50.920387 29692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:36365 every 8 connection(s)
I20260807 18:27:50.921612 29551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:27:50.922624 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 29551
I20260807 18:27:50.922853 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:27:50.923723 27655 external_mini_cluster.cc:1428] Setting key 69bd721bb44892a1b86ed49cd88f321c
I20260807 18:27:50.924151 29693 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44077
I20260807 18:27:50.924400 29693 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:50.925122 29693 heartbeater.cc:511] Master 127.27.1.254:44077 requested a full tablet report, sending...
I20260807 18:27:50.927119 29478 ts_manager.cc:194] Registered new tserver with Master: b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193:36365)
I20260807 18:27:50.928647 29478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:50609
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:27:50.970896 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44077
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:51.301225 29697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:51.301548 29697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:51.301724 29697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:51.309130 29697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:51.309296 29697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:51.309456 29697 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 18:27:51.309563 29697 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 18:27:51.309677 29697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:27:51.323539 29697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.1.254:44077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:51.324997 29697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:51.326293 29697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:51.339640 29702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:51.339648 29703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:51.339690 29705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:51.340858 29697 server_base.cc:1034] running on GCE node
I20260807 18:27:51.341408 29697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:51.342250 29697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:51.343441 29697 hybrid_clock.cc:648] HybridClock initialized: now 1786127271343391 us; error 38 us; skew 500 ppm
I20260807 18:27:51.343717 29697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:51 dist-test-slave-1xrh krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127271, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:51.346827 29697 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:27:51.348326 29697 webserver.cc:460] Webserver started at http://127.27.1.194:41421/ using document root <none> and password file <none>
I20260807 18:27:51.349009 29697 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:51.349109 29697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:51.349351 29697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:51.352222 29697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "41961d60a3844c989081d2c5764df957"
format_stamp: "Formatted at 2026-08-07 18:27:51 on dist-test-slave-1xrh"
server_key: "df447cc60bcbf442a15ff81c4415990d"
server_key_iv: "30792bf445250dc8bcc428968d037d6c"
server_key_version: "encryptionkey@0"
I20260807 18:27:51.352906 29697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "41961d60a3844c989081d2c5764df957"
format_stamp: "Formatted at 2026-08-07 18:27:51 on dist-test-slave-1xrh"
server_key: "df447cc60bcbf442a15ff81c4415990d"
server_key_iv: "30792bf445250dc8bcc428968d037d6c"
server_key_version: "encryptionkey@0"
I20260807 18:27:51.358001 29697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:27:51.361425 29713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:51.362710 29697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 18:27:51.362905 29697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "41961d60a3844c989081d2c5764df957"
format_stamp: "Formatted at 2026-08-07 18:27:51 on dist-test-slave-1xrh"
server_key: "df447cc60bcbf442a15ff81c4415990d"
server_key_iv: "30792bf445250dc8bcc428968d037d6c"
server_key_version: "encryptionkey@0"
I20260807 18:27:51.363109 29697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:51.383607 29697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:51.387739 29697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:51.388036 29697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:51.390746 29697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:51.390887 29697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:51.391008 29697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:51.391106 29697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 18:27:51 dist-test-slave-1xrh krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127271, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:51.415412 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:51.391421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:54295 (local address 127.27.1.254:44077)
0807 18:27:51.391644 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:51.391652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:51.393014 (+ 1362us) server_negotiation.cc:407] Connection header received
0807 18:27:51.395305 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:51.395312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:51.395408 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:51.395526 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:51.397770 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.398891 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.401590 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.402276 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.402428 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:51.407071 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:51.407076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:51.407079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:51.407115 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:51.411008 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:51.412297 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:51.412306 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:51.412312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:51.412491 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:51.413211 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:51.413214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:51.413217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:51.413521 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:51.413682 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:51.414494 (+ 812us) server_negotiation.cc:299] Negotiation successful
0807 18:27:51.414814 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 18:27:51.416013 29722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:51.392007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44077 (local address 127.27.1.194:54295)
0807 18:27:51.392829 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:51.392865 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:51.394891 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:51.395772 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:51.395781 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:51.396522 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:51.397514 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:51.397548 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.399084 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.399090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:51.401410 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:51.401431 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.402777 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.402785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:51.402981 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:51.403830 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:51.403878 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:51.406835 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:51.411247 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:51.411260 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:51.411281 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:51.412127 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:51.412648 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:51.412651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:51.412654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:51.413031 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:51.413660 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:51.413668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:51.414030 (+ 362us) client_negotiation.cc:769] Sending connection context
0807 18:27:51.414311 (+ 281us) client_negotiation.cc:240] Negotiation successful
0807 18:27:51.414712 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":368,"threads_started":1}
I20260807 18:27:51.437297 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:51.421724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:41555 (local address 127.27.1.254:44077)
0807 18:27:51.422308 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:51.422317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:51.422566 (+ 249us) server_negotiation.cc:407] Connection header received
0807 18:27:51.422754 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:51.422759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:51.422815 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:51.422914 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:51.424379 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.425510 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.427635 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.428093 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.428238 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:51.430982 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:51.430987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:51.430990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:51.431021 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:51.434466 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:51.435177 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:51.435184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:51.435188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:51.435330 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:51.435732 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:51.435739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:51.435742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:51.436071 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:51.436231 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:51.436700 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 18:27:51.436921 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260807 18:27:51.437399 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:51.421832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44077 (local address 127.27.1.194:41555)
0807 18:27:51.422454 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:51.422477 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:51.422631 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:51.422943 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:51.422947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:51.423588 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:51.424234 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:51.424246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.425656 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.425663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:51.427486 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:51.427498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.428488 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.428496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:51.428661 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:51.429532 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:51.429552 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:51.430727 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:51.434613 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:51.434623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:51.434627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:51.435028 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:51.435453 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:51.435458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:51.435465 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:51.435613 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:51.436197 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:51.436201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:51.436410 (+ 209us) client_negotiation.cc:769] Sending connection context
0807 18:27:51.436626 (+ 216us) client_negotiation.cc:240] Negotiation successful
0807 18:27:51.436952 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":480,"threads_started":1}
I20260807 18:27:51.446751 29697 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:43545
I20260807 18:27:51.446812 29837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:43545 every 8 connection(s)
I20260807 18:27:51.448114 29697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:27:51.450222 29838 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44077
I20260807 18:27:51.450462 29838 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:51.450654 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 29697
I20260807 18:27:51.450840 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:27:51.451346 29838 heartbeater.cc:511] Master 127.27.1.254:44077 requested a full tablet report, sending...
I20260807 18:27:51.451361 27655 external_mini_cluster.cc:1428] Setting key f56e56ec21e1de688b75d2366e3fb327
I20260807 18:27:51.452852 29477 ts_manager.cc:194] Registered new tserver with Master: 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545)
I20260807 18:27:51.453851 29477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:54295
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:27:51.496577 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44077
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:51.821563 29842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:51.821837 29842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:51.821939 29842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:51.829197 29842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:51.829310 29842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:51.829398 29842 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 18:27:51.829453 29842 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 18:27:51.829524 29842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:27:51.838963 29842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.1.254:44077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:51.840242 29842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:51.841554 29842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:51.855437 29847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:51.855463 29848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:51.855386 29850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:51.856149 29842 server_base.cc:1034] running on GCE node
I20260807 18:27:51.856837 29842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:51.857779 29842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:51.858987 29842 hybrid_clock.cc:648] HybridClock initialized: now 1786127271858943 us; error 33 us; skew 500 ppm
I20260807 18:27:51.859336 29842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:51 dist-test-slave-1xrh krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127271, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:51.862582 29842 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:27:51.864113 29842 webserver.cc:460] Webserver started at http://127.27.1.195:37483/ using document root <none> and password file <none>
I20260807 18:27:51.864763 29842 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:51.864886 29842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:51.865165 29842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:51.868469 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "7e89384b2522474c844ddd574d47fdd4"
format_stamp: "Formatted at 2026-08-07 18:27:51 on dist-test-slave-1xrh"
server_key: "234ff1c99fce2d2f67cea9739cae7425"
server_key_iv: "a8cda182141ce17d7afd64bf177fc54d"
server_key_version: "encryptionkey@0"
I20260807 18:27:51.869243 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "7e89384b2522474c844ddd574d47fdd4"
format_stamp: "Formatted at 2026-08-07 18:27:51 on dist-test-slave-1xrh"
server_key: "234ff1c99fce2d2f67cea9739cae7425"
server_key_iv: "a8cda182141ce17d7afd64bf177fc54d"
server_key_version: "encryptionkey@0"
I20260807 18:27:51.874781 29842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 18:27:51.878460 29857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:51.879899 29842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 18:27:51.880079 29842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "7e89384b2522474c844ddd574d47fdd4"
format_stamp: "Formatted at 2026-08-07 18:27:51 on dist-test-slave-1xrh"
server_key: "234ff1c99fce2d2f67cea9739cae7425"
server_key_iv: "a8cda182141ce17d7afd64bf177fc54d"
server_key_version: "encryptionkey@0"
I20260807 18:27:51.880276 29842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:51.900337 29842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:51.904675 29842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:51.904976 29842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:51.907591 29842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:27:51.907742 29842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:51.907897 29842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:27:51.907996 29842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 18:27:51 dist-test-slave-1xrh krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127271, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:51.932024 29693 heartbeater.cc:503] Master 127.27.1.254:44077 was elected leader, sending a full tablet report...
I20260807 18:27:51.935560 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:51.908933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44077 (local address 127.27.1.195:52489)
0807 18:27:51.909842 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:51.909902 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:51.912216 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:51.913149 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:51.913164 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:51.914177 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:51.915303 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:51.915334 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.916772 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.916781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:51.919758 (+ 2977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:51.919780 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.921399 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.921409 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:51.921785 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:51.922692 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:51.922732 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:51.926174 (+ 3442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:51.930312 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:51.930321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:51.930335 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:51.930904 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:51.931459 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:51.931466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:51.931471 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:51.931755 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:51.932541 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:51.932549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:51.933018 (+ 469us) client_negotiation.cc:769] Sending connection context
0807 18:27:51.933339 (+ 321us) client_negotiation.cc:240] Negotiation successful
0807 18:27:51.933938 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":429,"threads_started":1}
I20260807 18:27:51.936491 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:51.908402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:52489 (local address 127.27.1.254:44077)
0807 18:27:51.908600 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:51.908607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:51.910049 (+ 1442us) server_negotiation.cc:407] Connection header received
0807 18:27:51.912576 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:51.912582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:51.912682 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:51.912800 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:51.915540 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.916602 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.919986 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.920811 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.920944 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:51.926429 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:51.926435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:51.926441 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:51.926489 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:51.930182 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:51.931089 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:51.931095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:51.931098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:51.931270 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:51.931949 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:51.931953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:51.931956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:51.932314 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:51.932459 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:51.935466 (+ 3007us) server_negotiation.cc:299] Negotiation successful
0807 18:27:51.935980 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 18:27:51.958279 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:51.941453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:55099 (local address 127.27.1.254:44077)
0807 18:27:51.941639 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:51.941647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:51.942691 (+ 1044us) server_negotiation.cc:407] Connection header received
0807 18:27:51.942881 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:51.942884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:51.942942 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:51.943051 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:51.944315 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.945349 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.947569 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.948122 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.948318 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:51.951024 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:51.951031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:51.951035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:51.951072 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:51.955054 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:51.956058 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:51.956068 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:51.956071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:51.956228 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:51.956713 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:51.956720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:51.956725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:51.957094 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:51.957271 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:51.957658 (+ 387us) server_negotiation.cc:299] Negotiation successful
0807 18:27:51.957900 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:27:51.958388 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:51.941449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44077 (local address 127.27.1.195:55099)
0807 18:27:51.942554 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:51.942580 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:51.942755 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:51.943089 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:51.943095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:51.943794 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:51.944161 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:51.944171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.945519 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.945523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:51.947409 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:51.947424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:51.948602 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:51.948610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:51.948725 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:51.949548 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:51.949564 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:51.950800 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:51.955232 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:51.955243 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:51.955248 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:51.955908 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:51.956359 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:51.956367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:51.956374 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:51.956565 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:51.957224 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:51.957231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:51.957419 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 18:27:51.957561 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 18:27:51.957910 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":546,"threads_started":1}
I20260807 18:27:51.961793 29842 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:35537
I20260807 18:27:51.961861 29980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:35537 every 8 connection(s)
I20260807 18:27:51.963258 29842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:27:51.965240 29981 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44077
I20260807 18:27:51.965480 29981 heartbeater.cc:464] Registering TS with master...
I20260807 18:27:51.966033 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 29842
I20260807 18:27:51.966117 29981 heartbeater.cc:511] Master 127.27.1.254:44077 requested a full tablet report, sending...
I20260807 18:27:51.966223 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:27:51.966735 27655 external_mini_cluster.cc:1428] Setting key 0965dbe3b5e407054de48359b6845e0f
I20260807 18:27:51.967654 29477 ts_manager.cc:194] Registered new tserver with Master: 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537)
I20260807 18:27:51.968679 29477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:52489
I20260807 18:27:51.969833 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 18:27:51 dist-test-slave-1xrh krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 18:27:52 dist-test-slave-1xrh krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:52.021878 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.008338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.27.1.254:44077)
0807 18:27:52.008515 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:52.008519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:52.008858 (+ 339us) server_negotiation.cc:407] Connection header received
0807 18:27:52.009018 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:52.009023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:52.009090 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:52.009183 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:52.010543 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.011344 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.012573 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.013009 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.013096 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.016859 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:52.016864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:52.016868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:52.016905 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:52.019764 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:52.020316 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:52.020320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:52.020322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:52.020455 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:52.020760 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:52.020775 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:52.020776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:52.021021 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:52.021192 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:52.021445 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 18:27:52.021610 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 18:27:52.026033 29478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:27:52.032939 29478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:27:52.058101 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.045119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:43545 (local address 127.0.0.1:59664)
0807 18:27:52.046514 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:52.046562 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:52.046753 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:52.047760 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:52.047767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:52.047797 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:52.048226 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.048245 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.050453 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.050461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.053567 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.053589 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.057112 (+ 3523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.057121 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.057334 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.057387 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 18:27:52.057542 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 18:27:52.057740 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":11634,"threads_started":1}
I20260807 18:27:52.058076 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.045715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.27.1.194:43545)
0807 18:27:52.046472 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:52.046482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:52.046663 (+ 181us) server_negotiation.cc:407] Connection header received
0807 18:27:52.046967 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:52.046978 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:52.047268 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:52.047472 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:52.048480 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.050246 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.053814 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.055720 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.055910 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.056194 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:52.057585 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0807 18:27:52.057762 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":1799,"threads_started":1}
I20260807 18:27:52.061997 29764 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5d8ae9c0f74761b8e29b049b1274c1 (DEFAULT_TABLE table=test-table [id=0499bc1976f344309523a6c0e423c804]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:52.063632 29764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5d8ae9c0f74761b8e29b049b1274c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:52.064875 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.047097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:35537 (local address 127.0.0.1:55016)
0807 18:27:52.057119 (+ 10022us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:52.057141 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:52.057272 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:52.058545 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:52.058553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:52.058583 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:52.059061 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.059073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.060654 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.060661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.062015 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.062027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.064028 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.064036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.064211 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.064247 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 18:27:52.064339 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 18:27:52.064478 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9872,"mutex_wait_us":4,"thread_start_us":9252,"threads_started":1}
I20260807 18:27:52.065382 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.047334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55016 (local address 127.27.1.195:35537)
0807 18:27:52.057573 (+ 10239us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:52.057583 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:52.057617 (+ 34us) server_negotiation.cc:407] Connection header received
0807 18:27:52.057710 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:52.057717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:52.057921 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:52.058127 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:52.059366 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.060515 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.062190 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.063297 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.063443 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.063676 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:52.064642 (+ 966us) server_negotiation.cc:299] Negotiation successful
0807 18:27:52.064794 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10055,"thread_start_us":9827,"threads_started":1}
I20260807 18:27:52.066113 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.045269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36365 (local address 127.0.0.1:53214)
0807 18:27:52.056729 (+ 11460us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:52.056754 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:52.057436 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:52.058378 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:52.058382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:52.058599 (+ 217us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:52.059373 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.059384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.061371 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.061376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.063493 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.063507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.065284 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.065289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.065494 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.065537 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 18:27:52.065627 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 18:27:52.065761 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11283,"mutex_wait_us":5,"thread_start_us":875,"threads_started":1}
I20260807 18:27:52.066174 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.045532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.27.1.193:36365)
0807 18:27:52.046087 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:52.046099 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:52.056895 (+ 10796us) server_negotiation.cc:407] Connection header received
0807 18:27:52.057660 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:52.057669 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:52.057922 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:52.058102 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:52.059695 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.061165 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.063688 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.065102 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.065230 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.065405 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:52.065691 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 18:27:52.065854 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":242,"threads_started":1}
I20260807 18:27:52.069380 29908 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5d8ae9c0f74761b8e29b049b1274c1 (DEFAULT_TABLE table=test-table [id=0499bc1976f344309523a6c0e423c804]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:52.069416 29620 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5d8ae9c0f74761b8e29b049b1274c1 (DEFAULT_TABLE table=test-table [id=0499bc1976f344309523a6c0e423c804]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:52.070849 29620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5d8ae9c0f74761b8e29b049b1274c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:52.070849 29908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5d8ae9c0f74761b8e29b049b1274c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:52.075676 30000 tablet_bootstrap.cc:492] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: Bootstrap starting.
I20260807 18:27:52.079437 30000 tablet_bootstrap.cc:654] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:52.080443 30000 log.cc:824] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:52.082906 30000 tablet_bootstrap.cc:492] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: No bootstrap required, opened a new log
I20260807 18:27:52.083211 30000 ts_tablet_manager.cc:1397] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260807 18:27:52.085734 30002 tablet_bootstrap.cc:492] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: Bootstrap starting.
I20260807 18:27:52.086699 30003 tablet_bootstrap.cc:492] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: Bootstrap starting.
I20260807 18:27:52.089439 30002 tablet_bootstrap.cc:654] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:52.089690 30003 tablet_bootstrap.cc:654] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:52.090687 30002 log.cc:824] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:52.090754 30003 log.cc:824] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:52.094033 30002 tablet_bootstrap.cc:492] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: No bootstrap required, opened a new log
I20260807 18:27:52.094295 30002 ts_tablet_manager.cc:1397] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260807 18:27:52.095191 30000 raft_consensus.cc:348] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.095626 30000 raft_consensus.cc:374] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:52.095739 30000 raft_consensus.cc:729] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41961d60a3844c989081d2c5764df957, State: Initialized, Role: FOLLOWER
I20260807 18:27:52.096467 30000 consensus_queue.cc:260] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.097767 30003 tablet_bootstrap.cc:492] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: No bootstrap required, opened a new log
I20260807 18:27:52.098037 30003 ts_tablet_manager.cc:1397] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260807 18:27:52.099833 29838 heartbeater.cc:503] Master 127.27.1.254:44077 was elected leader, sending a full tablet report...
I20260807 18:27:52.101787 30000 ts_tablet_manager.cc:1428] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: Time spent starting tablet: real 0.018s user 0.008s sys 0.007s
I20260807 18:27:52.105202 30002 raft_consensus.cc:348] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.105556 30002 raft_consensus.cc:374] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:52.105660 30002 raft_consensus.cc:729] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b368a5ecd24349eeb7e48c438d852b96, State: Initialized, Role: FOLLOWER
I20260807 18:27:52.106384 30002 consensus_queue.cc:260] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.107287 30003 raft_consensus.cc:348] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.107709 30003 raft_consensus.cc:374] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:52.107822 30003 raft_consensus.cc:729] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e89384b2522474c844ddd574d47fdd4, State: Initialized, Role: FOLLOWER
I20260807 18:27:52.107929 30002 ts_tablet_manager.cc:1428] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: Time spent starting tablet: real 0.013s user 0.012s sys 0.003s
I20260807 18:27:52.108402 30003 consensus_queue.cc:260] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.109473 29981 heartbeater.cc:503] Master 127.27.1.254:44077 was elected leader, sending a full tablet report...
I20260807 18:27:52.110239 30003 ts_tablet_manager.cc:1428] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
W20260807 18:27:52.174198 29694 tablet.cc:2369] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:27:52.200136 29839 tablet.cc:2369] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:27:52.215054 29982 tablet.cc:2369] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:27:52.497776 30008 raft_consensus.cc:482] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:52.498090 30008 raft_consensus.cc:504] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.499850 30008 leader_election.cc:288] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545), b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193:36365)
I20260807 18:27:52.506728 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.500739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:38621 (local address 127.27.1.194:43545)
0807 18:27:52.500952 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:52.500959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:52.500979 (+ 20us) server_negotiation.cc:407] Connection header received
0807 18:27:52.501148 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:52.501156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:52.501238 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:52.501330 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:52.501935 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.502791 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.504798 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.505742 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.505922 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.505975 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:52.506346 (+ 371us) server_negotiation.cc:299] Negotiation successful
0807 18:27:52.506472 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 18:27:52.506842 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.500146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:43545 (local address 127.27.1.195:38621)
0807 18:27:52.500811 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:52.500834 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:52.500987 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:52.501365 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:52.501369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:52.501397 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:52.501708 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.501725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.502926 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.502931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.504628 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.504641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.505933 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.505941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.506158 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.506206 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 18:27:52.506305 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 18:27:52.506455 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":18,"thread_start_us":637,"threads_started":1}
I20260807 18:27:52.507282 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.500232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36365 (local address 127.27.1.195:54477)
0807 18:27:52.501091 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:52.501109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:52.501228 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:52.501602 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:52.501605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:52.501625 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:52.501989 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.502005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.503208 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.503212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.505010 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.505024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.506392 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.506399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.506574 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.506610 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 18:27:52.506705 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 18:27:52.506857 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":39,"thread_start_us":307,"threads_started":1}
I20260807 18:27:52.507563 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.500524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:54477 (local address 127.27.1.193:36365)
0807 18:27:52.500738 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:52.500747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:52.501215 (+ 468us) server_negotiation.cc:407] Connection header received
0807 18:27:52.501379 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:52.501384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:52.501474 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:52.501572 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:52.502199 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.503041 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.505162 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.506146 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.506318 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.506387 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:52.506765 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 18:27:52.507296 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:27:52.507941 29792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5d8ae9c0f74761b8e29b049b1274c1" candidate_uuid: "7e89384b2522474c844ddd574d47fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41961d60a3844c989081d2c5764df957" is_pre_election: true
I20260807 18:27:52.508457 29792 raft_consensus.cc:2454] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e89384b2522474c844ddd574d47fdd4 in term 0.
I20260807 18:27:52.508502 29648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5d8ae9c0f74761b8e29b049b1274c1" candidate_uuid: "7e89384b2522474c844ddd574d47fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b368a5ecd24349eeb7e48c438d852b96" is_pre_election: true
I20260807 18:27:52.509035 29648 raft_consensus.cc:2454] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e89384b2522474c844ddd574d47fdd4 in term 0.
I20260807 18:27:52.509116 29860 leader_election.cc:302] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41961d60a3844c989081d2c5764df957, 7e89384b2522474c844ddd574d47fdd4; no voters:
I20260807 18:27:52.509585 30008 raft_consensus.cc:2781] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:27:52.509755 30008 raft_consensus.cc:482] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:27:52.509871 30008 raft_consensus.cc:3037] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:52.512149 30008 raft_consensus.cc:504] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.512715 30008 leader_election.cc:288] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [CANDIDATE]: Term 1 election: Requested vote from peers 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545), b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193:36365)
I20260807 18:27:52.513171 29792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5d8ae9c0f74761b8e29b049b1274c1" candidate_uuid: "7e89384b2522474c844ddd574d47fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41961d60a3844c989081d2c5764df957"
I20260807 18:27:52.513171 29648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5d8ae9c0f74761b8e29b049b1274c1" candidate_uuid: "7e89384b2522474c844ddd574d47fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b368a5ecd24349eeb7e48c438d852b96"
I20260807 18:27:52.513368 29792 raft_consensus.cc:3037] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:52.513392 29648 raft_consensus.cc:3037] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:52.516685 29648 raft_consensus.cc:2454] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e89384b2522474c844ddd574d47fdd4 in term 1.
I20260807 18:27:52.516685 29792 raft_consensus.cc:2454] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e89384b2522474c844ddd574d47fdd4 in term 1.
I20260807 18:27:52.517082 29860 leader_election.cc:302] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41961d60a3844c989081d2c5764df957, 7e89384b2522474c844ddd574d47fdd4; no voters:
I20260807 18:27:52.517390 30008 raft_consensus.cc:2781] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:52.517795 30008 raft_consensus.cc:686] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 1 LEADER]: Becoming Leader. State: Replica: 7e89384b2522474c844ddd574d47fdd4, State: Running, Role: LEADER
I20260807 18:27:52.518241 30008 consensus_queue.cc:237] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.523228 29477 catalog_manager.cc:5495] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195). New cstate: current_term: 1 leader_uuid: "7e89384b2522474c844ddd574d47fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } health_report { overall_health: HEALTHY } } }
Aug 07 18:27:52 dist-test-slave-1xrh krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:52.654608 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.641387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44077 (local address 127.0.0.1:56482)
0807 18:27:52.642310 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:52.642327 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:52.642473 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:52.642784 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:52.642787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:52.643670 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:27:52.643890 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.643899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.644898 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.644901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.645834 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.645845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.646512 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.646516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:52.646642 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.647328 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:27:52.647345 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:27:52.649466 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:27:52.652467 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:52.652477 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:52.652483 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:52.652860 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:52.653182 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:27:52.653185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:27:52.653187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:27:52.653320 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:27:52.653779 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:27:52.653785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:27:52.653937 (+ 152us) client_negotiation.cc:769] Sending connection context
0807 18:27:52.654075 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 18:27:52.654270 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":620,"threads_started":1}
I20260807 18:27:52.654742 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.641399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.27.1.254:44077)
0807 18:27:52.641999 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:52.642004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:52.642428 (+ 424us) server_negotiation.cc:407] Connection header received
0807 18:27:52.642578 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:52.642582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:52.642639 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:52.642736 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:52.644022 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.644760 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.645970 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.646375 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.646465 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:52.649649 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:52.649654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:52.649657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:52.649690 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:52.652343 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:52.652969 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:52.652974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:52.652976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:52.653089 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:52.653409 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:52.653412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:52.653414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:52.653664 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:52.653784 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:52.654160 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 18:27:52.654397 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":486,"thread_start_us":335,"threads_started":1}
I20260807 18:27:52.659492 29477 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:56482:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 18:27:52.660424 29477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:27:52.666965 29764 tablet_service.cc:1467] Processing CreateTablet for tablet f493de0d70584e68b307667fce49cb07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b9d09fcfb3c4c819c9e198501f1334b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 18:27:52.666957 29620 tablet_service.cc:1467] Processing CreateTablet for tablet f493de0d70584e68b307667fce49cb07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b9d09fcfb3c4c819c9e198501f1334b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 18:27:52.667963 29620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f493de0d70584e68b307667fce49cb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:52.667963 29764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f493de0d70584e68b307667fce49cb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:52.667454 29908 tablet_service.cc:1467] Processing CreateTablet for tablet f493de0d70584e68b307667fce49cb07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b9d09fcfb3c4c819c9e198501f1334b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 18:27:52.668314 29908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f493de0d70584e68b307667fce49cb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:52.674585 30021 tablet_bootstrap.cc:492] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: Bootstrap starting.
I20260807 18:27:52.675367 30023 tablet_bootstrap.cc:492] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: Bootstrap starting.
I20260807 18:27:52.676206 30022 tablet_bootstrap.cc:492] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: Bootstrap starting.
I20260807 18:27:52.677268 30021 tablet_bootstrap.cc:654] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:52.677338 30023 tablet_bootstrap.cc:654] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:52.677780 30022 tablet_bootstrap.cc:654] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:52.679579 30021 tablet_bootstrap.cc:492] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: No bootstrap required, opened a new log
I20260807 18:27:52.679733 30021 ts_tablet_manager.cc:1397] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260807 18:27:52.679828 30023 tablet_bootstrap.cc:492] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: No bootstrap required, opened a new log
I20260807 18:27:52.679967 30023 ts_tablet_manager.cc:1397] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 18:27:52.680049 30022 tablet_bootstrap.cc:492] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: No bootstrap required, opened a new log
I20260807 18:27:52.680218 30022 ts_tablet_manager.cc:1397] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 18:27:52.680676 30023 raft_consensus.cc:348] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.680649 30021 raft_consensus.cc:348] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.680891 30023 raft_consensus.cc:374] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:52.680894 30021 raft_consensus.cc:374] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:52.680980 30023 raft_consensus.cc:729] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41961d60a3844c989081d2c5764df957, State: Initialized, Role: FOLLOWER
I20260807 18:27:52.680994 30021 raft_consensus.cc:729] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e89384b2522474c844ddd574d47fdd4, State: Initialized, Role: FOLLOWER
I20260807 18:27:52.680968 30022 raft_consensus.cc:348] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.681161 30022 raft_consensus.cc:374] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:52.681262 30022 raft_consensus.cc:729] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b368a5ecd24349eeb7e48c438d852b96, State: Initialized, Role: FOLLOWER
I20260807 18:27:52.681226 30023 consensus_queue.cc:260] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.681213 30021 consensus_queue.cc:260] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.681514 30022 consensus_queue.cc:260] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.681787 30021 ts_tablet_manager.cc:1428] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260807 18:27:52.681816 30008 tablet_replica.cc:439] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } } }
I20260807 18:27:52.682062 30008 tablet_replica.cc:442] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:52.682688 30027 tablet_replica.cc:439] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } } }
I20260807 18:27:52.683187 30027 tablet_replica.cc:442] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:52.683332 30023 ts_tablet_manager.cc:1428] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 18:27:52.683813 30028 tablet_replica.cc:439] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } } }
I20260807 18:27:52.684727 30028 tablet_replica.cc:442] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:52.683907 30022 ts_tablet_manager.cc:1428] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 18:27:52.992682 30028 raft_consensus.cc:482] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:52.992938 30028 raft_consensus.cc:504] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.994611 30028 leader_election.cc:288] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545), 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537)
I20260807 18:27:52.997635 30027 raft_consensus.cc:482] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:52.997893 30027 raft_consensus.cc:504] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:52.999693 30027 leader_election.cc:288] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193:36365), 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537)
I20260807 18:27:53.004454 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.994962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:35537 (local address 127.27.1.193:41581)
0807 18:27:52.995962 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:52.996007 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:52.996184 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:52.996846 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:52.996850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:52.996908 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:52.997534 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.997545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.998776 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.998781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.001325 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.001347 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.003391 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.003397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.003582 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.003705 (+ 123us) client_negotiation.cc:769] Sending connection context
0807 18:27:53.003813 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 18:27:53.003976 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":66,"spinlock_wait_cycles":27904,"thread_start_us":535,"threads_started":1}
I20260807 18:27:53.004858 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.995071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:47109 (local address 127.27.1.194:43545)
0807 18:27:52.995298 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:52.995305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:52.996476 (+ 1171us) server_negotiation.cc:407] Connection header received
0807 18:27:52.996600 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:52.996605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:52.996699 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:52.996802 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:52.997667 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.998875 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.002190 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.003561 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.003687 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.003756 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:53.004386 (+ 630us) server_negotiation.cc:299] Negotiation successful
0807 18:27:53.004541 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:27:53.005568 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.995473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:41581 (local address 127.27.1.195:35537)
0807 18:27:52.996545 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:52.996553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:52.996568 (+ 15us) server_negotiation.cc:407] Connection header received
0807 18:27:52.996632 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:52.996636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:52.996712 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:52.996811 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:52.997708 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.998636 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.001553 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.002957 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.003168 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.003255 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:53.004827 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0807 18:27:53.005185 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":784,"threads_started":1}
I20260807 18:27:53.005590 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.995085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:43545 (local address 127.27.1.193:47109)
0807 18:27:52.996335 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:52.996354 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:52.996487 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:52.996846 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:52.996850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:52.996910 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:52.997446 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:52.997468 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:52.999010 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:52.999015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.002011 (+ 2996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.002030 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.003975 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.003980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.004198 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.004228 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 18:27:53.005050 (+ 822us) client_negotiation.cc:240] Negotiation successful
0807 18:27:53.005184 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":783,"spinlock_wait_cycles":9856,"thread_start_us":797,"threads_started":1}
I20260807 18:27:53.006448 29792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f493de0d70584e68b307667fce49cb07" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41961d60a3844c989081d2c5764df957" is_pre_election: true
I20260807 18:27:53.006732 29792 raft_consensus.cc:2454] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 0.
I20260807 18:27:53.007350 29571 leader_election.cc:302] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41961d60a3844c989081d2c5764df957, b368a5ecd24349eeb7e48c438d852b96; no voters:
I20260807 18:27:53.008061 30028 raft_consensus.cc:2781] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:27:53.007992 29935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f493de0d70584e68b307667fce49cb07" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e89384b2522474c844ddd574d47fdd4" is_pre_election: true
I20260807 18:27:53.008193 30028 raft_consensus.cc:482] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:27:53.008297 30028 raft_consensus.cc:3037] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:53.008360 29935 raft_consensus.cc:2454] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 0.
I20260807 18:27:53.009975 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.000282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:56333 (local address 127.27.1.193:36365)
0807 18:27:53.000507 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:53.000515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:53.001091 (+ 576us) server_negotiation.cc:407] Connection header received
0807 18:27:53.001352 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:53.001357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:53.001460 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:53.001540 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:53.002643 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.004240 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.007500 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.008826 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.008969 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.009038 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:53.009544 (+ 506us) server_negotiation.cc:299] Negotiation successful
0807 18:27:53.009679 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 18:27:53.010001 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:52.999905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36365 (local address 127.27.1.194:56333)
0807 18:27:53.001002 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:53.001021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:53.001189 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:53.001759 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:53.001763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:53.001790 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:53.002431 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.002450 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.004568 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.004576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.007301 (+ 2725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.007315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.009223 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.009231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.009370 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.009393 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 18:27:53.009519 (+ 126us) client_negotiation.cc:240] Negotiation successful
0807 18:27:53.009653 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":790,"threads_started":1}
I20260807 18:27:53.010798 29648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f493de0d70584e68b307667fce49cb07" candidate_uuid: "41961d60a3844c989081d2c5764df957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b368a5ecd24349eeb7e48c438d852b96" is_pre_election: true
I20260807 18:27:53.011929 30028 raft_consensus.cc:504] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.012509 29648 raft_consensus.cc:2379] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41961d60a3844c989081d2c5764df957 in current term 1: Already voted for candidate b368a5ecd24349eeb7e48c438d852b96 in this term.
I20260807 18:27:53.012799 30028 leader_election.cc:288] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 election: Requested vote from peers 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545), 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537)
I20260807 18:27:53.013073 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.001136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:35537 (local address 127.27.1.194:50961)
0807 18:27:53.001722 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:53.001741 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:53.001859 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:53.006498 (+ 4639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:53.006502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:53.006530 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:27:53.006993 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.007009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.008422 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.008427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.010501 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.010516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.012233 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.012242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.012449 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.012490 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 18:27:53.012591 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 18:27:53.012719 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":493,"threads_started":1}
I20260807 18:27:53.013384 29935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f493de0d70584e68b307667fce49cb07" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e89384b2522474c844ddd574d47fdd4"
I20260807 18:27:53.013536 29935 raft_consensus.cc:3037] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:53.013430 29792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f493de0d70584e68b307667fce49cb07" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41961d60a3844c989081d2c5764df957"
I20260807 18:27:53.013671 29792 raft_consensus.cc:3037] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:53.015717 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.001470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:50961 (local address 127.27.1.195:35537)
0807 18:27:53.006110 (+ 4640us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:53.006117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:53.006138 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:27:53.006257 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:53.006269 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:53.006353 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:53.006479 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:27:53.007210 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.008246 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.010691 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.011990 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.012148 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.012219 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:53.015246 (+ 3027us) server_negotiation.cc:299] Negotiation successful
0807 18:27:53.015387 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4514,"thread_start_us":5478,"threads_started":1}
I20260807 18:27:53.016413 29935 raft_consensus.cc:2454] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 1.
I20260807 18:27:53.016477 29792 raft_consensus.cc:2454] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 1.
I20260807 18:27:53.016391 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f493de0d70584e68b307667fce49cb07" candidate_uuid: "41961d60a3844c989081d2c5764df957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e89384b2522474c844ddd574d47fdd4" is_pre_election: true
I20260807 18:27:53.016779 29934 raft_consensus.cc:2379] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41961d60a3844c989081d2c5764df957 in current term 1: Already voted for candidate b368a5ecd24349eeb7e48c438d852b96 in this term.
I20260807 18:27:53.016889 29570 leader_election.cc:302] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e89384b2522474c844ddd574d47fdd4, b368a5ecd24349eeb7e48c438d852b96; no voters:
I20260807 18:27:53.017215 30028 raft_consensus.cc:2781] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:53.017509 29715 leader_election.cc:302] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41961d60a3844c989081d2c5764df957; no voters: 7e89384b2522474c844ddd574d47fdd4, b368a5ecd24349eeb7e48c438d852b96
I20260807 18:27:53.017763 30028 raft_consensus.cc:686] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [term 1 LEADER]: Becoming Leader. State: Replica: b368a5ecd24349eeb7e48c438d852b96, State: Running, Role: LEADER
I20260807 18:27:53.018189 30027 raft_consensus.cc:2726] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 18:27:53.018601 30028 consensus_queue.cc:237] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.021278 30035 tablet_replica.cc:439] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: TxnStatusTablet state changed. Reason: New leader b368a5ecd24349eeb7e48c438d852b96. Latest consensus state: current_term: 1 leader_uuid: "b368a5ecd24349eeb7e48c438d852b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } } }
I20260807 18:27:53.021672 30035 tablet_replica.cc:442] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: This TxnStatusTablet replica's current role is: LEADER
I20260807 18:27:53.022543 30037 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 18:27:53.022810 30037 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 18:27:53.023315 29478 catalog_manager.cc:5495] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193). New cstate: current_term: 1 leader_uuid: "b368a5ecd24349eeb7e48c438d852b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } health_report { overall_health: UNKNOWN } } }
I20260807 18:27:53.049611 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.043385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.27.1.193:36365)
0807 18:27:53.043592 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:53.043600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:53.043742 (+ 142us) server_negotiation.cc:407] Connection header received
0807 18:27:53.044034 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:53.044038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:53.044123 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:53.044243 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:53.044660 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.045450 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.047437 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.047874 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.047994 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.048557 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:53.049047 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:53.049175 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:53.049330 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 18:27:53.049406 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 18:27:53.049799 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.043368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36365 (local address 127.0.0.1:53224)
0807 18:27:53.043642 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:53.043664 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:53.043844 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:53.044303 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:53.044308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:53.044323 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 18:27:53.044524 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.044536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.045603 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.045609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.047260 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.047275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.048064 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.048073 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.048292 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.048366 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 18:27:53.049184 (+ 818us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 18:27:53.049207 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 18:27:53.049305 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 18:27:53.049475 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 18:27:53.057202 29934 raft_consensus.cc:1261] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Refusing update from remote peer b368a5ecd24349eeb7e48c438d852b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:53.057305 29792 raft_consensus.cc:1261] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Refusing update from remote peer b368a5ecd24349eeb7e48c438d852b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:53.058082 30035 consensus_queue.cc:1035] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:53.058440 30028 consensus_queue.cc:1035] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:53.068246 30008 tablet_replica.cc:439] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: TxnStatusTablet state changed. Reason: New leader b368a5ecd24349eeb7e48c438d852b96. Latest consensus state: current_term: 1 leader_uuid: "b368a5ecd24349eeb7e48c438d852b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } } }
I20260807 18:27:53.068569 30008 tablet_replica.cc:442] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:53.070798 30027 tablet_replica.cc:439] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: TxnStatusTablet state changed. Reason: New leader b368a5ecd24349eeb7e48c438d852b96. Latest consensus state: current_term: 1 leader_uuid: "b368a5ecd24349eeb7e48c438d852b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } } }
I20260807 18:27:53.071096 30027 tablet_replica.cc:442] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:53.071350 30040 mvcc.cc:205] Tried to move back new op lower bound from 7315977310433177600 to 7315977310288277504. Current Snapshot: MvccSnapshot[applied={T|T < 7315977310433177600}]
I20260807 18:27:53.074051 30008 consensus_queue.cc:1035] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:53.088181 30008 tablet_replica.cc:439] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b368a5ecd24349eeb7e48c438d852b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } } }
I20260807 18:27:53.088568 30008 tablet_replica.cc:442] T f493de0d70584e68b307667fce49cb07 P 7e89384b2522474c844ddd574d47fdd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:53.092324 30052 tablet_replica.cc:439] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b368a5ecd24349eeb7e48c438d852b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } } }
I20260807 18:27:53.092849 30052 tablet_replica.cc:442] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: This TxnStatusTablet replica's current role is: LEADER
I20260807 18:27:53.096633 30035 tablet_replica.cc:439] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b368a5ecd24349eeb7e48c438d852b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } } }
I20260807 18:27:53.096895 30035 tablet_replica.cc:442] T f493de0d70584e68b307667fce49cb07 P b368a5ecd24349eeb7e48c438d852b96: This TxnStatusTablet replica's current role is: LEADER
I20260807 18:27:53.107167 30008 consensus_queue.cc:1035] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:53.120733 30027 tablet_replica.cc:439] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b368a5ecd24349eeb7e48c438d852b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } } }
I20260807 18:27:53.121018 30027 tablet_replica.cc:442] T f493de0d70584e68b307667fce49cb07 P 41961d60a3844c989081d2c5764df957: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:27:53.131574 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.124051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.27.1.195:35537)
0807 18:27:53.124222 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:53.124228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:53.125228 (+ 1000us) server_negotiation.cc:407] Connection header received
0807 18:27:53.125574 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:53.125581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:53.125691 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:53.125840 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:53.126499 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.127717 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.129085 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.129727 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.129871 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.130262 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:53.130948 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:53.131093 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:53.131179 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 18:27:53.131266 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 18:27:53.147799 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.141722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36365 (local address 127.27.1.195:33323)
0807 18:27:53.142413 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:53.142433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:53.142540 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:53.142888 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:53.142890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:53.142899 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 18:27:53.143217 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.143226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.144663 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.144666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.145749 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.145761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.146767 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.146771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.146880 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.146931 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 18:27:53.147358 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 18:27:53.147367 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 18:27:53.147434 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 18:27:53.147535 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":368,"threads_started":1}
I20260807 18:27:53.147908 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.141754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:33323 (local address 127.27.1.193:36365)
0807 18:27:53.141949 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:53.141956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:53.142514 (+ 558us) server_negotiation.cc:407] Connection header received
0807 18:27:53.142672 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:53.142678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:53.142772 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:53.142872 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:53.143357 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.144518 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.145890 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.146596 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.146729 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.147115 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:53.147263 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:53.147357 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:53.147508 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 18:27:53.147587 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:27:53.192406 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.185957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:54271 (local address 127.27.1.195:35537)
0807 18:27:53.186130 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:53.186136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:53.186438 (+ 302us) server_negotiation.cc:407] Connection header received
0807 18:27:53.186585 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:53.186594 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:53.186681 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:53.186791 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:53.187405 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.188505 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.190199 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.190884 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.191015 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.191590 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:53.191753 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:53.191848 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:53.192032 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 18:27:53.192117 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 18:27:53.192549 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.185756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:35537 (local address 127.27.1.193:54271)
0807 18:27:53.186333 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:53.186351 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:53.186466 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:53.186838 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:53.186843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:53.186860 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 18:27:53.187229 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.187245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.188650 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.188655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.190004 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.190017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.191087 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.191093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.191301 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.191403 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 18:27:53.191876 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 18:27:53.191894 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 18:27:53.191995 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 18:27:53.192144 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":286,"threads_started":1}
I20260807 18:27:53.239658 29478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:53.239993 29478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:53.244263 29478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1: Sending DeleteTablet for 3 replicas of tablet 2d5d8ae9c0f74761b8e29b049b1274c1
I20260807 18:27:53.247512 29764 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5d8ae9c0f74761b8e29b049b1274c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:53 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:59664
I20260807 18:27:53.249127 29908 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5d8ae9c0f74761b8e29b049b1274c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:53 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:55016
I20260807 18:27:53.256949 29620 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5d8ae9c0f74761b8e29b049b1274c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:53 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:53214
I20260807 18:27:53.258921 29478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:27:53.262660 29478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:27:53.263625 30067 tablet_replica.cc:330] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: stopping tablet replica
I20260807 18:27:53.264130 30067 raft_consensus.cc:2229] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:53.264505 30067 raft_consensus.cc:2258] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:53.266384 30069 tablet_replica.cc:330] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: stopping tablet replica
I20260807 18:27:53.266579 30067 ts_tablet_manager.cc:1905] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:53.267226 30069 raft_consensus.cc:2229] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:53.267730 30069 raft_consensus.cc:2258] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:53.269898 30069 ts_tablet_manager.cc:1905] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:53.271484 30068 tablet_replica.cc:330] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: stopping tablet replica
I20260807 18:27:53.271973 30068 raft_consensus.cc:2229] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:27:53.272667 30068 raft_consensus.cc:2258] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:53.273588 29764 tablet_service.cc:1467] Processing CreateTablet for tablet 27773cbc5e8f4f8cba9877e638d40171 (DEFAULT_TABLE table=test-table [id=20be0fe14ea34e2d96fbf2018080e5d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:53.273619 29620 tablet_service.cc:1467] Processing CreateTablet for tablet 27773cbc5e8f4f8cba9877e638d40171 (DEFAULT_TABLE table=test-table [id=20be0fe14ea34e2d96fbf2018080e5d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:53.274029 29764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27773cbc5e8f4f8cba9877e638d40171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:53.274046 29620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27773cbc5e8f4f8cba9877e638d40171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:53.279464 29908 tablet_service.cc:1467] Processing CreateTablet for tablet 27773cbc5e8f4f8cba9877e638d40171 (DEFAULT_TABLE table=test-table [id=20be0fe14ea34e2d96fbf2018080e5d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:53.279881 29908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27773cbc5e8f4f8cba9877e638d40171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:53.281922 30068 ts_tablet_manager.cc:1905] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:53.282212 30069 ts_tablet_manager.cc:1918] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:27:53.282421 30069 log.cc:1196] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/2d5d8ae9c0f74761b8e29b049b1274c1
I20260807 18:27:53.283020 30069 ts_tablet_manager.cc:1939] T 2d5d8ae9c0f74761b8e29b049b1274c1 P b368a5ecd24349eeb7e48c438d852b96: Deleting consensus metadata
I20260807 18:27:53.288415 29463 catalog_manager.cc:4841] TS b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193:36365): tablet 2d5d8ae9c0f74761b8e29b049b1274c1 (table test-table [id=0499bc1976f344309523a6c0e423c804]) successfully deleted
I20260807 18:27:53.289813 30070 tablet_bootstrap.cc:492] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957: Bootstrap starting.
I20260807 18:27:53.295609 30071 tablet_bootstrap.cc:492] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96: Bootstrap starting.
I20260807 18:27:53.298267 30071 tablet_bootstrap.cc:654] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:53.299358 30070 tablet_bootstrap.cc:654] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:53.301570 30068 ts_tablet_manager.cc:1918] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:27:53.303323 30068 log.cc:1196] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/2d5d8ae9c0f74761b8e29b049b1274c1
I20260807 18:27:53.304401 30068 ts_tablet_manager.cc:1939] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 7e89384b2522474c844ddd574d47fdd4: Deleting consensus metadata
I20260807 18:27:53.306653 30067 ts_tablet_manager.cc:1918] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:27:53.306833 30067 log.cc:1196] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/2d5d8ae9c0f74761b8e29b049b1274c1
I20260807 18:27:53.307415 30067 ts_tablet_manager.cc:1939] T 2d5d8ae9c0f74761b8e29b049b1274c1 P 41961d60a3844c989081d2c5764df957: Deleting consensus metadata
I20260807 18:27:53.307719 30070 tablet_bootstrap.cc:492] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957: No bootstrap required, opened a new log
I20260807 18:27:53.307922 30070 ts_tablet_manager.cc:1397] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s
I20260807 18:27:53.308792 30070 raft_consensus.cc:348] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.309218 29464 catalog_manager.cc:4841] TS 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545): tablet 2d5d8ae9c0f74761b8e29b049b1274c1 (table test-table [id=0499bc1976f344309523a6c0e423c804]) successfully deleted
I20260807 18:27:53.309262 30070 raft_consensus.cc:374] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:53.309432 30070 raft_consensus.cc:729] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41961d60a3844c989081d2c5764df957, State: Initialized, Role: FOLLOWER
I20260807 18:27:53.309782 30070 consensus_queue.cc:260] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.310984 29463 catalog_manager.cc:4841] TS 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537): tablet 2d5d8ae9c0f74761b8e29b049b1274c1 (table test-table [id=0499bc1976f344309523a6c0e423c804]) successfully deleted
I20260807 18:27:53.314095 30070 ts_tablet_manager.cc:1428] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260807 18:27:53.314816 30073 tablet_bootstrap.cc:492] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4: Bootstrap starting.
I20260807 18:27:53.317137 30073 tablet_bootstrap.cc:654] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:53.318674 30071 tablet_bootstrap.cc:492] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96: No bootstrap required, opened a new log
I20260807 18:27:53.319303 30071 ts_tablet_manager.cc:1397] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20260807 18:27:53.320236 30071 raft_consensus.cc:348] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.320523 30071 raft_consensus.cc:374] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:53.320641 30071 raft_consensus.cc:729] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b368a5ecd24349eeb7e48c438d852b96, State: Initialized, Role: FOLLOWER
I20260807 18:27:53.320904 30071 consensus_queue.cc:260] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.323982 30071 ts_tablet_manager.cc:1428] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 18:27:53.324026 30073 tablet_bootstrap.cc:492] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4: No bootstrap required, opened a new log
I20260807 18:27:53.324317 30073 ts_tablet_manager.cc:1397] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260807 18:27:53.326555 30073 raft_consensus.cc:348] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.326817 30073 raft_consensus.cc:374] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:53.326924 30073 raft_consensus.cc:729] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e89384b2522474c844ddd574d47fdd4, State: Initialized, Role: FOLLOWER
I20260807 18:27:53.327251 30073 consensus_queue.cc:260] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.328073 30073 ts_tablet_manager.cc:1428] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260807 18:27:53.435935 30035 raft_consensus.cc:482] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:53.436205 30035 raft_consensus.cc:504] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.437076 30035 leader_election.cc:288] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545), 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537)
I20260807 18:27:53.437747 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27773cbc5e8f4f8cba9877e638d40171" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41961d60a3844c989081d2c5764df957" is_pre_election: true
I20260807 18:27:53.438056 29791 raft_consensus.cc:2454] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 0.
I20260807 18:27:53.437968 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27773cbc5e8f4f8cba9877e638d40171" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e89384b2522474c844ddd574d47fdd4" is_pre_election: true
I20260807 18:27:53.438301 29934 raft_consensus.cc:2454] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 0.
I20260807 18:27:53.438588 29571 leader_election.cc:302] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41961d60a3844c989081d2c5764df957, b368a5ecd24349eeb7e48c438d852b96; no voters:
I20260807 18:27:53.439042 30035 raft_consensus.cc:2781] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:27:53.439172 30035 raft_consensus.cc:482] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:27:53.439299 30035 raft_consensus.cc:3037] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:53.442731 30035 raft_consensus.cc:504] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.443507 30035 leader_election.cc:288] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 election: Requested vote from peers 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545), 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537)
I20260807 18:27:53.444056 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27773cbc5e8f4f8cba9877e638d40171" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41961d60a3844c989081d2c5764df957"
I20260807 18:27:53.444284 29791 raft_consensus.cc:3037] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:53.444252 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27773cbc5e8f4f8cba9877e638d40171" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e89384b2522474c844ddd574d47fdd4"
I20260807 18:27:53.444500 29934 raft_consensus.cc:3037] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:53.447499 29791 raft_consensus.cc:2454] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 1.
I20260807 18:27:53.448035 29571 leader_election.cc:302] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41961d60a3844c989081d2c5764df957, b368a5ecd24349eeb7e48c438d852b96; no voters:
I20260807 18:27:53.448230 29934 raft_consensus.cc:2454] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 1.
I20260807 18:27:53.448640 30035 raft_consensus.cc:2781] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:53.449157 30035 raft_consensus.cc:686] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 1 LEADER]: Becoming Leader. State: Replica: b368a5ecd24349eeb7e48c438d852b96, State: Running, Role: LEADER
I20260807 18:27:53.449440 30035 consensus_queue.cc:237] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:53.468713 29478 catalog_manager.cc:5495] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193). New cstate: current_term: 1 leader_uuid: "b368a5ecd24349eeb7e48c438d852b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } health_report { overall_health: UNKNOWN } } }
I20260807 18:27:53.506726 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.501307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.27.1.193:36365)
0807 18:27:53.501466 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:53.501470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:53.501546 (+ 76us) server_negotiation.cc:407] Connection header received
0807 18:27:53.501744 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:53.501747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:53.501811 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:53.501922 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:53.502347 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.503246 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.504759 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.505413 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.505565 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.506021 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:53.506127 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:53.506241 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:53.506372 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 18:27:53.506452 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 18:27:53.520186 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.515140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:35257 (local address 127.27.1.193:36365)
0807 18:27:53.515280 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:53.515284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:53.515312 (+ 28us) server_negotiation.cc:407] Connection header received
0807 18:27:53.515390 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:53.515395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:53.515531 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:53.515628 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:53.516071 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.516843 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.518242 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.518800 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.518935 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.519382 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:53.519488 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:53.519585 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:53.519700 (+ 115us) server_negotiation.cc:299] Negotiation successful
0807 18:27:53.519770 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":53}
I20260807 18:27:53.520208 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:53.514898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36365 (local address 127.27.1.193:35257)
0807 18:27:53.515092 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 18:27:53.515112 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:27:53.515285 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:27:53.515640 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:27:53.515644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:27:53.515654 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 18:27:53.515938 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.515948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.516974 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.516979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.518104 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:27:53.518116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:27:53.518947 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:27:53.518952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:27:53.519164 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:53.519209 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 18:27:53.519587 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 18:27:53.519599 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 18:27:53.519679 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 18:27:53.519837 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":28}
I20260807 18:27:53.530696 29934 raft_consensus.cc:1261] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Refusing update from remote peer b368a5ecd24349eeb7e48c438d852b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:53.530802 29791 raft_consensus.cc:1261] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Refusing update from remote peer b368a5ecd24349eeb7e48c438d852b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:53.531388 30028 consensus_queue.cc:1035] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:53.531822 30028 consensus_queue.cc:1035] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:53.551867 30042 mvcc.cc:205] Tried to move back new op lower bound from 7315977312375971840 to 7315977312052174848. Current Snapshot: MvccSnapshot[applied={T|T < 7315977312375971840}]
W20260807 18:27:53.567852 30040 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 18:27:55.166265 29476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:55.166491 29476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:55.170514 29476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1: Sending DeleteTablet for 3 replicas of tablet 27773cbc5e8f4f8cba9877e638d40171
I20260807 18:27:55.171291 29764 tablet_service.cc:1514] Processing DeleteTablet for tablet 27773cbc5e8f4f8cba9877e638d40171 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:55 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:59664
I20260807 18:27:55.171294 29619 tablet_service.cc:1514] Processing DeleteTablet for tablet 27773cbc5e8f4f8cba9877e638d40171 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:55 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:53214
I20260807 18:27:55.171746 29908 tablet_service.cc:1514] Processing DeleteTablet for tablet 27773cbc5e8f4f8cba9877e638d40171 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:55 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:55016
I20260807 18:27:55.172117 30104 tablet_replica.cc:330] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96: stopping tablet replica
I20260807 18:27:55.172204 30103 tablet_replica.cc:330] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957: stopping tablet replica
I20260807 18:27:55.172431 30104 raft_consensus.cc:2229] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:27:55.172468 30103 raft_consensus.cc:2229] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:55.172756 30103 raft_consensus.cc:2258] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:55.173135 30104 raft_consensus.cc:2258] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:55.174465 30104 ts_tablet_manager.cc:1905] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:55.181681 30103 ts_tablet_manager.cc:1905] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:55.182343 30104 ts_tablet_manager.cc:1918] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 18:27:55.182544 30104 log.cc:1196] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/27773cbc5e8f4f8cba9877e638d40171
I20260807 18:27:55.182755 30105 tablet_replica.cc:330] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4: stopping tablet replica
I20260807 18:27:55.183027 30105 raft_consensus.cc:2229] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:55.183292 30104 ts_tablet_manager.cc:1939] T 27773cbc5e8f4f8cba9877e638d40171 P b368a5ecd24349eeb7e48c438d852b96: Deleting consensus metadata
I20260807 18:27:55.183357 30105 raft_consensus.cc:2258] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:55.184523 30105 ts_tablet_manager.cc:1905] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:55.184711 29463 catalog_manager.cc:4841] TS b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193:36365): tablet 27773cbc5e8f4f8cba9877e638d40171 (table test-table [id=20be0fe14ea34e2d96fbf2018080e5d0]) successfully deleted
I20260807 18:27:55.187692 30103 ts_tablet_manager.cc:1918] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 18:27:55.187852 30103 log.cc:1196] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/27773cbc5e8f4f8cba9877e638d40171
I20260807 18:27:55.188412 30103 ts_tablet_manager.cc:1939] T 27773cbc5e8f4f8cba9877e638d40171 P 41961d60a3844c989081d2c5764df957: Deleting consensus metadata
I20260807 18:27:55.189437 29464 catalog_manager.cc:4841] TS 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545): tablet 27773cbc5e8f4f8cba9877e638d40171 (table test-table [id=20be0fe14ea34e2d96fbf2018080e5d0]) successfully deleted
I20260807 18:27:55.191264 30105 ts_tablet_manager.cc:1918] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 18:27:55.191413 30105 log.cc:1196] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/27773cbc5e8f4f8cba9877e638d40171
I20260807 18:27:55.191874 30105 ts_tablet_manager.cc:1939] T 27773cbc5e8f4f8cba9877e638d40171 P 7e89384b2522474c844ddd574d47fdd4: Deleting consensus metadata
I20260807 18:27:55.193025 29463 catalog_manager.cc:4841] TS 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537): tablet 27773cbc5e8f4f8cba9877e638d40171 (table test-table [id=20be0fe14ea34e2d96fbf2018080e5d0]) successfully deleted
Aug 07 18:27:55 dist-test-slave-1xrh krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127275, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 18:27:55 dist-test-slave-1xrh krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127275, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:55.240557 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:55.218691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.27.1.254:44077)
0807 18:27:55.219389 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:55.219394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:55.219416 (+ 22us) server_negotiation.cc:407] Connection header received
0807 18:27:55.219556 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:55.219559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:55.219624 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:55.219766 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:55.221187 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:55.221929 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:55.223246 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:55.223804 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:55.223924 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:55.235045 (+ 11121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:55.235050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:55.235054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:55.235089 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:55.237770 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:55.238483 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:55.238489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:55.238493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:55.238640 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:55.239045 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:55.239050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:55.239053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:55.239404 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:55.239670 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:55.239964 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 18:27:55.240185 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":386,"threads_started":1}
I20260807 18:27:55.244349 29476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:27:55.244997 29476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:27:55.252481 29619 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef3c76f18e34390938e4c8c63a41551 (DEFAULT_TABLE table=test-table [id=e881644782744a3996f6c86b4a99b89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:55.252696 29764 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef3c76f18e34390938e4c8c63a41551 (DEFAULT_TABLE table=test-table [id=e881644782744a3996f6c86b4a99b89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:55.253099 29619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef3c76f18e34390938e4c8c63a41551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:55.253114 29764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef3c76f18e34390938e4c8c63a41551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:55.256810 29908 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef3c76f18e34390938e4c8c63a41551 (DEFAULT_TABLE table=test-table [id=e881644782744a3996f6c86b4a99b89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:55.257284 29908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef3c76f18e34390938e4c8c63a41551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:55.267671 30115 tablet_bootstrap.cc:492] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96: Bootstrap starting.
I20260807 18:27:55.268021 30114 tablet_bootstrap.cc:492] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957: Bootstrap starting.
I20260807 18:27:55.269196 30116 tablet_bootstrap.cc:492] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4: Bootstrap starting.
I20260807 18:27:55.269935 30115 tablet_bootstrap.cc:654] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:55.270612 30114 tablet_bootstrap.cc:654] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:55.270921 30116 tablet_bootstrap.cc:654] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:55.278475 30115 tablet_bootstrap.cc:492] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96: No bootstrap required, opened a new log
I20260807 18:27:55.278712 30115 ts_tablet_manager.cc:1397] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260807 18:27:55.279438 30114 tablet_bootstrap.cc:492] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957: No bootstrap required, opened a new log
I20260807 18:27:55.279595 30116 tablet_bootstrap.cc:492] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4: No bootstrap required, opened a new log
I20260807 18:27:55.279817 30116 ts_tablet_manager.cc:1397] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260807 18:27:55.279784 30115 raft_consensus.cc:348] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.280138 30115 raft_consensus.cc:374] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:55.280236 30115 raft_consensus.cc:729] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b368a5ecd24349eeb7e48c438d852b96, State: Initialized, Role: FOLLOWER
I20260807 18:27:55.279592 30114 ts_tablet_manager.cc:1397] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260807 18:27:55.280516 30115 consensus_queue.cc:260] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.280732 30116 raft_consensus.cc:348] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.281185 30116 raft_consensus.cc:374] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:55.281278 30115 ts_tablet_manager.cc:1428] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 18:27:55.281334 30116 raft_consensus.cc:729] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e89384b2522474c844ddd574d47fdd4, State: Initialized, Role: FOLLOWER
I20260807 18:27:55.281478 30114 raft_consensus.cc:348] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.281739 30114 raft_consensus.cc:374] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:55.281606 30116 consensus_queue.cc:260] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.281831 30114 raft_consensus.cc:729] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41961d60a3844c989081d2c5764df957, State: Initialized, Role: FOLLOWER
I20260807 18:27:55.282081 30114 consensus_queue.cc:260] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.282938 30114 ts_tablet_manager.cc:1428] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 18:27:55.283659 30116 ts_tablet_manager.cc:1428] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260807 18:27:55.425698 30121 raft_consensus.cc:482] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:55.425973 30121 raft_consensus.cc:504] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.426841 30121 leader_election.cc:288] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193:36365), 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545)
I20260807 18:27:55.427431 29648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef3c76f18e34390938e4c8c63a41551" candidate_uuid: "7e89384b2522474c844ddd574d47fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b368a5ecd24349eeb7e48c438d852b96" is_pre_election: true
I20260807 18:27:55.427510 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef3c76f18e34390938e4c8c63a41551" candidate_uuid: "7e89384b2522474c844ddd574d47fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41961d60a3844c989081d2c5764df957" is_pre_election: true
I20260807 18:27:55.427711 29648 raft_consensus.cc:2454] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e89384b2522474c844ddd574d47fdd4 in term 0.
I20260807 18:27:55.427843 29791 raft_consensus.cc:2454] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e89384b2522474c844ddd574d47fdd4 in term 0.
I20260807 18:27:55.428272 29859 leader_election.cc:302] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e89384b2522474c844ddd574d47fdd4, b368a5ecd24349eeb7e48c438d852b96; no voters:
I20260807 18:27:55.428579 30121 raft_consensus.cc:2781] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:27:55.428710 30121 raft_consensus.cc:482] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:27:55.428826 30121 raft_consensus.cc:3037] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:55.430948 30121 raft_consensus.cc:504] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.431529 30121 leader_election.cc:288] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [CANDIDATE]: Term 1 election: Requested vote from peers b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193:36365), 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545)
I20260807 18:27:55.432088 29648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef3c76f18e34390938e4c8c63a41551" candidate_uuid: "7e89384b2522474c844ddd574d47fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b368a5ecd24349eeb7e48c438d852b96"
I20260807 18:27:55.432111 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef3c76f18e34390938e4c8c63a41551" candidate_uuid: "7e89384b2522474c844ddd574d47fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41961d60a3844c989081d2c5764df957"
I20260807 18:27:55.432328 29791 raft_consensus.cc:3037] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:55.432353 29648 raft_consensus.cc:3037] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:55.435616 29648 raft_consensus.cc:2454] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e89384b2522474c844ddd574d47fdd4 in term 1.
I20260807 18:27:55.435632 29791 raft_consensus.cc:2454] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e89384b2522474c844ddd574d47fdd4 in term 1.
I20260807 18:27:55.436091 29860 leader_election.cc:302] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41961d60a3844c989081d2c5764df957, 7e89384b2522474c844ddd574d47fdd4; no voters:
I20260807 18:27:55.436455 30121 raft_consensus.cc:2781] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:55.436885 30121 raft_consensus.cc:686] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 1 LEADER]: Becoming Leader. State: Replica: 7e89384b2522474c844ddd574d47fdd4, State: Running, Role: LEADER
I20260807 18:27:55.437227 30121 consensus_queue.cc:237] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.440099 29478 catalog_manager.cc:5495] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195). New cstate: current_term: 1 leader_uuid: "7e89384b2522474c844ddd574d47fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } health_report { overall_health: HEALTHY } } }
I20260807 18:27:55.470194 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:55.464061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.27.1.195:35537)
0807 18:27:55.464978 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:55.464984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:55.465060 (+ 76us) server_negotiation.cc:407] Connection header received
0807 18:27:55.465120 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:55.465125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:55.465197 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:55.465295 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:55.465693 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:55.466920 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:55.468292 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:55.468931 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:55.469072 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:55.469436 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:55.469573 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:55.469657 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:55.469758 (+ 101us) server_negotiation.cc:299] Negotiation successful
0807 18:27:55.469832 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":659,"threads_started":1}
I20260807 18:27:55.487882 29648 raft_consensus.cc:1261] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Refusing update from remote peer 7e89384b2522474c844ddd574d47fdd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:55.487964 29791 raft_consensus.cc:1261] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Refusing update from remote peer 7e89384b2522474c844ddd574d47fdd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:55.488499 30123 consensus_queue.cc:1035] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:55.488862 30121 consensus_queue.cc:1035] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:55.494443 30085 mvcc.cc:205] Tried to move back new op lower bound from 7315977320393699328 to 7315977320193105920. Current Snapshot: MvccSnapshot[applied={T|T < 7315977320393699328}]
I20260807 18:27:55.530038 29478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:55.530272 29478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:55.533078 29478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1: Sending DeleteTablet for 3 replicas of tablet 3ef3c76f18e34390938e4c8c63a41551
I20260807 18:27:55.533855 29908 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef3c76f18e34390938e4c8c63a41551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:55 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:55016
I20260807 18:27:55.533789 29619 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef3c76f18e34390938e4c8c63a41551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:55 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:53214
I20260807 18:27:55.533860 29764 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef3c76f18e34390938e4c8c63a41551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:55 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:59664
I20260807 18:27:55.534123 30104 tablet_replica.cc:330] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96: stopping tablet replica
I20260807 18:27:55.534123 30105 tablet_replica.cc:330] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4: stopping tablet replica
I20260807 18:27:55.534124 30103 tablet_replica.cc:330] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957: stopping tablet replica
I20260807 18:27:55.534319 30104 raft_consensus.cc:2229] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:55.534346 30103 raft_consensus.cc:2229] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:55.534320 30105 raft_consensus.cc:2229] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:27:55.534516 30104 raft_consensus.cc:2258] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:55.534564 30103 raft_consensus.cc:2258] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:55.534706 30105 raft_consensus.cc:2258] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:55.534878 29478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:27:55.535595 29478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:27:55.535971 30103 ts_tablet_manager.cc:1905] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:55.536010 30104 ts_tablet_manager.cc:1905] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:55.537458 30105 ts_tablet_manager.cc:1905] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:55.541236 30104 ts_tablet_manager.cc:1918] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:27:55.541428 30104 log.cc:1196] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/3ef3c76f18e34390938e4c8c63a41551
I20260807 18:27:55.541941 30104 ts_tablet_manager.cc:1939] T 3ef3c76f18e34390938e4c8c63a41551 P b368a5ecd24349eeb7e48c438d852b96: Deleting consensus metadata
I20260807 18:27:55.543030 29463 catalog_manager.cc:4841] TS b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193:36365): tablet 3ef3c76f18e34390938e4c8c63a41551 (table test-table [id=e881644782744a3996f6c86b4a99b89c]) successfully deleted
I20260807 18:27:55.544026 30105 ts_tablet_manager.cc:1918] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:27:55.544147 30103 ts_tablet_manager.cc:1918] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:27:55.544193 30105 log.cc:1196] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/3ef3c76f18e34390938e4c8c63a41551
I20260807 18:27:55.544395 30103 log.cc:1196] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/3ef3c76f18e34390938e4c8c63a41551
I20260807 18:27:55.544775 30105 ts_tablet_manager.cc:1939] T 3ef3c76f18e34390938e4c8c63a41551 P 7e89384b2522474c844ddd574d47fdd4: Deleting consensus metadata
I20260807 18:27:55.544962 30103 ts_tablet_manager.cc:1939] T 3ef3c76f18e34390938e4c8c63a41551 P 41961d60a3844c989081d2c5764df957: Deleting consensus metadata
I20260807 18:27:55.545728 29463 catalog_manager.cc:4841] TS 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537): tablet 3ef3c76f18e34390938e4c8c63a41551 (table test-table [id=e881644782744a3996f6c86b4a99b89c]) successfully deleted
I20260807 18:27:55.546301 29464 catalog_manager.cc:4841] TS 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545): tablet 3ef3c76f18e34390938e4c8c63a41551 (table test-table [id=e881644782744a3996f6c86b4a99b89c]) successfully deleted
I20260807 18:27:55.547009 29619 tablet_service.cc:1467] Processing CreateTablet for tablet 75721075b4964c4daa55c4c4f2f26210 (DEFAULT_TABLE table=test-table [id=8a62de58f63347feb7ff71f9be36fcd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:55.547111 29764 tablet_service.cc:1467] Processing CreateTablet for tablet 75721075b4964c4daa55c4c4f2f26210 (DEFAULT_TABLE table=test-table [id=8a62de58f63347feb7ff71f9be36fcd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:55.547320 29908 tablet_service.cc:1467] Processing CreateTablet for tablet 75721075b4964c4daa55c4c4f2f26210 (DEFAULT_TABLE table=test-table [id=8a62de58f63347feb7ff71f9be36fcd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:27:55.547458 29619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75721075b4964c4daa55c4c4f2f26210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:55.547583 29764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75721075b4964c4daa55c4c4f2f26210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:55.547761 29908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75721075b4964c4daa55c4c4f2f26210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:55.551865 30114 tablet_bootstrap.cc:492] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957: Bootstrap starting.
I20260807 18:27:55.553268 30114 tablet_bootstrap.cc:654] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:55.554428 30116 tablet_bootstrap.cc:492] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4: Bootstrap starting.
I20260807 18:27:55.555190 30115 tablet_bootstrap.cc:492] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96: Bootstrap starting.
I20260807 18:27:55.556082 30114 tablet_bootstrap.cc:492] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957: No bootstrap required, opened a new log
I20260807 18:27:55.556208 30116 tablet_bootstrap.cc:654] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:55.556253 30114 ts_tablet_manager.cc:1397] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 18:27:55.556955 30115 tablet_bootstrap.cc:654] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:55.557042 30114 raft_consensus.cc:348] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.557488 30114 raft_consensus.cc:374] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:55.557638 30114 raft_consensus.cc:729] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41961d60a3844c989081d2c5764df957, State: Initialized, Role: FOLLOWER
I20260807 18:27:55.558068 30114 consensus_queue.cc:260] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.559971 30115 tablet_bootstrap.cc:492] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96: No bootstrap required, opened a new log
I20260807 18:27:55.559990 30114 ts_tablet_manager.cc:1428] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260807 18:27:55.560143 30115 ts_tablet_manager.cc:1397] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 18:27:55.560985 30115 raft_consensus.cc:348] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.561508 30115 raft_consensus.cc:374] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:55.561616 30115 raft_consensus.cc:729] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b368a5ecd24349eeb7e48c438d852b96, State: Initialized, Role: FOLLOWER
I20260807 18:27:55.562007 30116 tablet_bootstrap.cc:492] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4: No bootstrap required, opened a new log
I20260807 18:27:55.562029 30115 consensus_queue.cc:260] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.562234 30116 ts_tablet_manager.cc:1397] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260807 18:27:55.562681 30115 ts_tablet_manager.cc:1428] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 18:27:55.563015 30116 raft_consensus.cc:348] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.563349 30116 raft_consensus.cc:374] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:55.563444 30116 raft_consensus.cc:729] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e89384b2522474c844ddd574d47fdd4, State: Initialized, Role: FOLLOWER
I20260807 18:27:55.563692 30116 consensus_queue.cc:260] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.564355 30116 ts_tablet_manager.cc:1428] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260807 18:27:55.728605 30084 raft_consensus.cc:482] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:27:55.728888 30084 raft_consensus.cc:504] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.729717 30084 leader_election.cc:288] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545), 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537)
I20260807 18:27:55.730331 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75721075b4964c4daa55c4c4f2f26210" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41961d60a3844c989081d2c5764df957" is_pre_election: true
I20260807 18:27:55.730329 29935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75721075b4964c4daa55c4c4f2f26210" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e89384b2522474c844ddd574d47fdd4" is_pre_election: true
I20260807 18:27:55.730582 29791 raft_consensus.cc:2454] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 0.
I20260807 18:27:55.730630 29935 raft_consensus.cc:2454] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 0.
I20260807 18:27:55.731034 29571 leader_election.cc:302] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41961d60a3844c989081d2c5764df957, b368a5ecd24349eeb7e48c438d852b96; no voters:
I20260807 18:27:55.731390 30084 raft_consensus.cc:2781] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:27:55.731496 30084 raft_consensus.cc:482] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:27:55.731565 30084 raft_consensus.cc:3037] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:55.733585 30084 raft_consensus.cc:504] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.734292 30084 leader_election.cc:288] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 election: Requested vote from peers 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545), 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537)
I20260807 18:27:55.734831 29935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75721075b4964c4daa55c4c4f2f26210" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e89384b2522474c844ddd574d47fdd4"
I20260807 18:27:55.734844 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75721075b4964c4daa55c4c4f2f26210" candidate_uuid: "b368a5ecd24349eeb7e48c438d852b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41961d60a3844c989081d2c5764df957"
I20260807 18:27:55.735049 29791 raft_consensus.cc:3037] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:55.735072 29935 raft_consensus.cc:3037] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:55.738090 29791 raft_consensus.cc:2454] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 1.
I20260807 18:27:55.738078 29935 raft_consensus.cc:2454] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b368a5ecd24349eeb7e48c438d852b96 in term 1.
I20260807 18:27:55.738505 29570 leader_election.cc:302] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e89384b2522474c844ddd574d47fdd4, b368a5ecd24349eeb7e48c438d852b96; no voters:
I20260807 18:27:55.738826 30084 raft_consensus.cc:2781] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:55.738982 30084 raft_consensus.cc:686] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 1 LEADER]: Becoming Leader. State: Replica: b368a5ecd24349eeb7e48c438d852b96, State: Running, Role: LEADER
I20260807 18:27:55.739295 30084 consensus_queue.cc:237] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } }
I20260807 18:27:55.742345 29478 catalog_manager.cc:5495] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193). New cstate: current_term: 1 leader_uuid: "b368a5ecd24349eeb7e48c438d852b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b368a5ecd24349eeb7e48c438d852b96" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 } health_report { overall_health: UNKNOWN } } }
I20260807 18:27:55.806725 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:55.801237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.27.1.193:36365)
0807 18:27:55.801747 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:55.801752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:55.801771 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:27:55.801820 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:55.801823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:55.801880 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:55.802003 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:27:55.802427 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:55.803386 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:55.804841 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:55.805284 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:55.805423 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:55.805875 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:27:55.805993 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:27:55.806090 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:55.806255 (+ 165us) server_negotiation.cc:299] Negotiation successful
0807 18:27:55.806333 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":273,"threads_started":1}
I20260807 18:27:55.815346 29791 raft_consensus.cc:1261] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Refusing update from remote peer b368a5ecd24349eeb7e48c438d852b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:55.815356 29934 raft_consensus.cc:1261] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Refusing update from remote peer b368a5ecd24349eeb7e48c438d852b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:27:55.815979 30097 consensus_queue.cc:1035] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41961d60a3844c989081d2c5764df957" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:27:55.816278 30084 consensus_queue.cc:1035] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e89384b2522474c844ddd574d47fdd4" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 35537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 18:27:55.834830 30082 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 18:27:57.433341 29478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:57.433662 29478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:27:57.438155 29478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e188b524b9b7470baf52cdbb1f0feca1: Sending DeleteTablet for 3 replicas of tablet 75721075b4964c4daa55c4c4f2f26210
I20260807 18:27:57.438961 29619 tablet_service.cc:1514] Processing DeleteTablet for tablet 75721075b4964c4daa55c4c4f2f26210 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:57 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:53214
I20260807 18:27:57.439208 29764 tablet_service.cc:1514] Processing DeleteTablet for tablet 75721075b4964c4daa55c4c4f2f26210 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:57 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:59664
I20260807 18:27:57.439356 29908 tablet_service.cc:1514] Processing DeleteTablet for tablet 75721075b4964c4daa55c4c4f2f26210 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:27:57 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:55016
I20260807 18:27:57.440333 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29551
I20260807 18:27:57.454883 30162 tablet_replica.cc:330] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4: stopping tablet replica
I20260807 18:27:57.455154 30162 raft_consensus.cc:2229] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:57.455261 30161 tablet_replica.cc:330] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957: stopping tablet replica
I20260807 18:27:57.455406 30162 raft_consensus.cc:2258] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:57.455587 30161 raft_consensus.cc:2229] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:27:57.455981 30161 raft_consensus.cc:2258] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:27:57.457299 30161 ts_tablet_manager.cc:1905] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:57.457522 30162 ts_tablet_manager.cc:1905] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:57.465898 30161 ts_tablet_manager.cc:1918] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 18:27:57.466149 30161 log.cc:1196] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/75721075b4964c4daa55c4c4f2f26210
I20260807 18:27:57.466596 30160 tablet_replica.cc:330] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96: stopping tablet replica
I20260807 18:27:57.466795 30160 raft_consensus.cc:2229] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:27:57.467008 30161 ts_tablet_manager.cc:1939] T 75721075b4964c4daa55c4c4f2f26210 P 41961d60a3844c989081d2c5764df957: Deleting consensus metadata
I20260807 18:27:57.467190 30160 raft_consensus.cc:2258] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 07 18:27:57 dist-test-slave-1xrh krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127275, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:27:57.467645 30162 ts_tablet_manager.cc:1918] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 18:27:57.467789 30162 log.cc:1196] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/75721075b4964c4daa55c4c4f2f26210
I20260807 18:27:57.468274 30162 ts_tablet_manager.cc:1939] T 75721075b4964c4daa55c4c4f2f26210 P 7e89384b2522474c844ddd574d47fdd4: Deleting consensus metadata
I20260807 18:27:57.468389 30160 ts_tablet_manager.cc:1905] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:27:57.468454 29464 catalog_manager.cc:4841] TS 41961d60a3844c989081d2c5764df957 (127.27.1.194:43545): tablet 75721075b4964c4daa55c4c4f2f26210 (table test-table [id=8a62de58f63347feb7ff71f9be36fcd1]) successfully deleted
I20260807 18:27:57.469422 29463 catalog_manager.cc:4841] TS 7e89384b2522474c844ddd574d47fdd4 (127.27.1.195:35537): tablet 75721075b4964c4daa55c4c4f2f26210 (table test-table [id=8a62de58f63347feb7ff71f9be36fcd1]) successfully deleted
I20260807 18:27:57.474013 30160 ts_tablet_manager.cc:1918] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 18:27:57.474179 30160 log.cc:1196] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/75721075b4964c4daa55c4c4f2f26210
I20260807 18:27:57.474782 30160 ts_tablet_manager.cc:1939] T 75721075b4964c4daa55c4c4f2f26210 P b368a5ecd24349eeb7e48c438d852b96: Deleting consensus metadata
I20260807 18:27:57.475934 29463 catalog_manager.cc:4841] TS b368a5ecd24349eeb7e48c438d852b96 (127.27.1.193:36365): tablet 75721075b4964c4daa55c4c4f2f26210 (table test-table [id=8a62de58f63347feb7ff71f9be36fcd1]) successfully deleted
I20260807 18:27:57.477203 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:57.443428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.27.1.193:36365)
0807 18:27:57.455215 (+ 11787us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:57.455225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:57.455257 (+ 32us) server_negotiation.cc:407] Connection header received
0807 18:27:57.455449 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:57.455455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:57.455565 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:57.457612 (+ 2047us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:57.457787 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:57.458989 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:57.461698 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:57.462343 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:57.462483 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:57.469304 (+ 6821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:57.469325 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:57.469353 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:57.469404 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:57.473969 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:57.474900 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:57.474913 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:57.474920 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:57.475113 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:57.475593 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:57.475598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:57.475601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:57.475989 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:57.476305 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:57.476627 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 18:27:57.476853 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11478,"thread_start_us":11579,"threads_started":1}
W20260807 18:27:57.477921 29688 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53256
W20260807 18:27:57.478709 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29551: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:27:57.478925 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 29551
I20260807 18:27:57.511507 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29697
Aug 07 18:27:57 dist-test-slave-1xrh krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127275, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:27:57.526746 30165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:57.512292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.27.1.194:43545)
0807 18:27:57.512819 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:57.512823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:57.512838 (+ 15us) server_negotiation.cc:407] Connection header received
0807 18:27:57.512891 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:57.512895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:57.512956 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:57.513087 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:57.514423 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:57.515373 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:57.516630 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:57.517059 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:57.517206 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:57.521515 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:57.521527 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:57.521542 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:57.521577 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:57.524424 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:57.525032 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:57.525039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:57.525042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:57.525170 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:57.525498 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:57.525501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:57.525503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:57.525832 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:57.526063 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:57.526305 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 18:27:57.526468 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":256,"threads_started":1}
W20260807 18:27:57.527465 29833 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59678
W20260807 18:27:57.528127 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29697: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:27:57.528293 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 29697
I20260807 18:27:57.549521 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29842
Aug 07 18:27:57 dist-test-slave-1xrh krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127275, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:27:57.564738 30166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:57.550292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.27.1.195:35537)
0807 18:27:57.550804 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:57.550809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:57.550826 (+ 17us) server_negotiation.cc:407] Connection header received
0807 18:27:57.550883 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:57.550886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:57.550966 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:57.551058 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:57.552429 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:57.553221 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:57.554402 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:57.554779 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:57.554867 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:57.559019 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:57.559033 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:57.559051 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:57.559091 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:57.561997 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:57.562662 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:57.562671 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:57.562673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:57.562796 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:57.563200 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:57.563206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:57.563211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:57.563574 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:57.563887 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:57.564195 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 18:27:57.564430 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":279,"threads_started":1}
W20260807 18:27:57.565394 29976 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55046
W20260807 18:27:57.565944 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29842: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:27:57.566087 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 29842
I20260807 18:27:57.592752 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 29444
I20260807 18:27:57.593937 29520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:27:57.711755 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 29444
2026-08-07T18:27:57Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9246 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:27:57 dist-test-slave-1xrh krb5kdc[30170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:27:57 dist-test-slave-1xrh krb5kdc[30170](info): set up 2 sockets
Aug 07 18:27:57 dist-test-slave-1xrh krb5kdc[30170](info): commencing operation
krb5kdc: starting...
W20260807 18:27:58.892725 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:27:59 dist-test-slave-1xrh krb5kdc[30170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:27:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:27:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:27:59.111550 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44749
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:27:59.446859 30183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:27:59.447261 30183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:27:59.447432 30183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:27:59.454651 30183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:27:59.454746 30183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:27:59.454838 30183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:27:59.454954 30183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:27:59.455016 30183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:27:59.464952 30183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:44749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:27:59.466271 30183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:27:59.467653 30183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:27:59.479880 30188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:59.480028 30189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:27:59.479987 30191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:27:59.480582 30183 server_base.cc:1034] running on GCE node
I20260807 18:27:59.481383 30183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:27:59.482754 30183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:27:59.483960 30183 hybrid_clock.cc:648] HybridClock initialized: now 1786127279483908 us; error 42 us; skew 500 ppm
I20260807 18:27:59.484325 30183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:27:59 dist-test-slave-1xrh krb5kdc[30170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127279, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:27:59.487583 30183 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:27:59.489197 30183 webserver.cc:460] Webserver started at http://127.27.1.254:40125/ using document root <none> and password file <none>
I20260807 18:27:59.489903 30183 fs_manager.cc:359] Metadata directory not provided
I20260807 18:27:59.490017 30183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:27:59.490373 30183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:27:59.493422 30183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "cbfe06a5955842c485167a9b96710cba"
format_stamp: "Formatted at 2026-08-07 18:27:59 on dist-test-slave-1xrh"
server_key: "11f62e59d93dfd616ac9b5128b1f078d"
server_key_iv: "f33fa54fae404bbe1db0bfc635c5d80e"
server_key_version: "encryptionkey@0"
I20260807 18:27:59.494210 30183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "cbfe06a5955842c485167a9b96710cba"
format_stamp: "Formatted at 2026-08-07 18:27:59 on dist-test-slave-1xrh"
server_key: "11f62e59d93dfd616ac9b5128b1f078d"
server_key_iv: "f33fa54fae404bbe1db0bfc635c5d80e"
server_key_version: "encryptionkey@0"
I20260807 18:27:59.499475 30183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 18:27:59.502749 30198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:27:59.504145 30183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:27:59.504340 30183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "cbfe06a5955842c485167a9b96710cba"
format_stamp: "Formatted at 2026-08-07 18:27:59 on dist-test-slave-1xrh"
server_key: "11f62e59d93dfd616ac9b5128b1f078d"
server_key_iv: "f33fa54fae404bbe1db0bfc635c5d80e"
server_key_version: "encryptionkey@0"
I20260807 18:27:59.504545 30183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:27:59.527784 30183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:27:59.531685 30183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:27:59.531982 30183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:27:59.551079 30183 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:44749
I20260807 18:27:59.551225 30251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:44749 every 8 connection(s)
I20260807 18:27:59.552449 30183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:27:59.556088 30252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:27:59.560778 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 30183
I20260807 18:27:59.561081 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:27:59.561910 27655 external_mini_cluster.cc:1428] Setting key 3bdc0473f317d74b40e39f38a1352da7
I20260807 18:27:59.567636 30252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba: Bootstrap starting.
I20260807 18:27:59.571178 30252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba: Neither blocks nor log segments found. Creating new log.
I20260807 18:27:59.572381 30252 log.cc:824] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba: Log is configured to *not* fsync() on all Append() calls
I20260807 18:27:59.575182 30252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba: No bootstrap required, opened a new log
Aug 07 18:27:59 dist-test-slave-1xrh krb5kdc[30170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:27:59.585711 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:27:59.563909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.27.1.254:44749)
0807 18:27:59.564698 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0807 18:27:59.564709 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:27:59.564748 (+ 39us) server_negotiation.cc:407] Connection header received
0807 18:27:59.566556 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:27:59.566577 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:27:59.566927 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:27:59.567309 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:27:59.569037 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:59.570329 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:59.572081 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:27:59.572827 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:27:59.573070 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:27:59.576952 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:27:59.576964 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:27:59.576982 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:27:59.577030 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:27:59.581131 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:59.581830 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:59.581839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:59.581847 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:59.582049 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:27:59.582459 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:27:59.582477 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:27:59.582480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:27:59.583185 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:27:59.583515 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0807 18:27:59.583874 (+ 359us) server_negotiation.cc:299] Negotiation successful
0807 18:27:59.584331 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":388,"threads_started":1}
I20260807 18:27:59.585866 30252 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe06a5955842c485167a9b96710cba" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44749 } }
I20260807 18:27:59.586366 30252 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:27:59.586495 30252 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfe06a5955842c485167a9b96710cba, State: Initialized, Role: FOLLOWER
I20260807 18:27:59.587172 30252 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe06a5955842c485167a9b96710cba" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44749 } }
I20260807 18:27:59.587407 30252 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:27:59.587572 30252 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:27:59.587764 30252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:27:59.590188 30252 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe06a5955842c485167a9b96710cba" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44749 } }
I20260807 18:27:59.590682 30252 leader_election.cc:302] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbfe06a5955842c485167a9b96710cba; no voters:
I20260807 18:27:59.591310 30252 leader_election.cc:288] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:27:59.591491 30257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:27:59.592527 30257 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [term 1 LEADER]: Becoming Leader. State: Replica: cbfe06a5955842c485167a9b96710cba, State: Running, Role: LEADER
I20260807 18:27:59.593030 30257 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe06a5955842c485167a9b96710cba" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44749 } }
I20260807 18:27:59.593502 30252 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:27:59.595760 30259 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbfe06a5955842c485167a9b96710cba. Latest consensus state: current_term: 1 leader_uuid: "cbfe06a5955842c485167a9b96710cba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe06a5955842c485167a9b96710cba" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44749 } } }
I20260807 18:27:59.596150 30259 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [sys.catalog]: This master's current role is: LEADER
I20260807 18:27:59.595866 30258 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbfe06a5955842c485167a9b96710cba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe06a5955842c485167a9b96710cba" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44749 } } }
I20260807 18:27:59.596526 30258 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba [sys.catalog]: This master's current role is: LEADER
I20260807 18:27:59.600931 30262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:27:59.606160 30262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:27:59.615137 30262 catalog_manager.cc:1347] Generated new cluster ID: 8c65418743d748f0b41b50d84a51c122
I20260807 18:27:59.615257 30262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:27:59.625094 30262 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:27:59.626906 30262 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:27:59.638358 30262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbfe06a5955842c485167a9b96710cba: Generated new TSK 0
I20260807 18:27:59.639246 30262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:27:59.694940 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44749
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:00.031898 30279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:00.032192 30279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:00.032361 30279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:00.039599 30279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:00.039706 30279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:00.039921 30279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:00.049618 30279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:00.050948 30279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:00.052320 30279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:00.065825 30284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:00.065860 30287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:00.065917 30285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:00.066895 30279 server_base.cc:1034] running on GCE node
I20260807 18:28:00.067588 30279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:00.068563 30279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:00.069809 30279 hybrid_clock.cc:648] HybridClock initialized: now 1786127280069767 us; error 52 us; skew 500 ppm
I20260807 18:28:00.070159 30279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:00 dist-test-slave-1xrh krb5kdc[30170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:00.073412 30279 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:28:00.075165 30279 webserver.cc:460] Webserver started at http://127.27.1.193:44197/ using document root <none> and password file <none>
I20260807 18:28:00.075907 30279 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:00.076072 30279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:00.076407 30279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:00.079481 30279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "62c5eda71a364d528e1af4a0f167e04b"
format_stamp: "Formatted at 2026-08-07 18:28:00 on dist-test-slave-1xrh"
server_key: "e78dba69f33eeeeffe65b356f9b02942"
server_key_iv: "9345a9644b2100ec5fb0e90485de14fd"
server_key_version: "encryptionkey@0"
I20260807 18:28:00.080330 30279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "62c5eda71a364d528e1af4a0f167e04b"
format_stamp: "Formatted at 2026-08-07 18:28:00 on dist-test-slave-1xrh"
server_key: "e78dba69f33eeeeffe65b356f9b02942"
server_key_iv: "9345a9644b2100ec5fb0e90485de14fd"
server_key_version: "encryptionkey@0"
I20260807 18:28:00.085924 30279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 18:28:00.089663 30294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:00.091022 30279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 18:28:00.091246 30279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "62c5eda71a364d528e1af4a0f167e04b"
format_stamp: "Formatted at 2026-08-07 18:28:00 on dist-test-slave-1xrh"
server_key: "e78dba69f33eeeeffe65b356f9b02942"
server_key_iv: "9345a9644b2100ec5fb0e90485de14fd"
server_key_version: "encryptionkey@0"
I20260807 18:28:00.091461 30279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:00.115515 30279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:00.119602 30279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:00.119903 30279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:00.120909 30279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:00.122257 30279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:00.122370 30279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:00.122519 30279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:00.122594 30279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:00.156102 30279 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:41647
I20260807 18:28:00.156143 30410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:41647 every 8 connection(s)
I20260807 18:28:00.157387 30279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:00.158144 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 30279
I20260807 18:28:00.158421 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:28:00.159003 27655 external_mini_cluster.cc:1428] Setting key cda79043d914c4c5d44f997cd39a0368
Aug 07 18:28:00 dist-test-slave-1xrh krb5kdc[30170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:00.181442 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:00.160242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:35095 (local address 127.27.1.254:44749)
0807 18:28:00.160727 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:00.160733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:00.161816 (+ 1083us) server_negotiation.cc:407] Connection header received
0807 18:28:00.164777 (+ 2961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:00.164781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:00.164847 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:00.164942 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:00.167076 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:00.167841 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:00.170006 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:00.170780 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:00.170924 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:00.174286 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:00.174295 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:00.174303 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:00.174350 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:00.178064 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:00.178850 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:00.178858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:00.178864 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:00.179020 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:00.179482 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:00.179488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:00.179492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:00.179891 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:00.180070 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:00.180761 (+ 691us) server_negotiation.cc:299] Negotiation successful
0807 18:28:00.181002 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":233,"threads_started":1}
I20260807 18:28:00.182070 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:00.160720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44749 (local address 127.27.1.193:35095)
0807 18:28:00.161627 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:00.161680 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:00.164459 (+ 2779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:00.165223 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:00.165236 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:00.166047 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:00.166875 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:00.166905 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:00.168014 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:00.168022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:00.169861 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:00.169873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:00.170998 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:00.171003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:00.171246 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:00.171939 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:00.171961 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:00.174056 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:00.178217 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:00.178224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:00.178237 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:00.178705 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:00.179176 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:00.179180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:00.179182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:00.179359 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:00.180023 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:00.180029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:00.180354 (+ 325us) client_negotiation.cc:769] Sending connection context
0807 18:28:00.180586 (+ 232us) client_negotiation.cc:240] Negotiation successful
0807 18:28:00.180951 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":342,"threads_started":1}
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
I20260807 18:28:00.183682 30411 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44749
I20260807 18:28:00.184010 30411 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:00.184803 30411 heartbeater.cc:511] Master 127.27.1.254:44749 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
I20260807 18:28:00.187265 30215 ts_manager.cc:194] Registered new tserver with Master: 62c5eda71a364d528e1af4a0f167e04b (127.27.1.193:41647)
I20260807 18:28:00.188701 30215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:35095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:28:00.215422 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44749
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:00.547487 30417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:00.547817 30417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:00.547989 30417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:00.555734 30417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:00.555939 30417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:00.556126 30417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:00.566038 30417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:00.567370 30417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:00.568694 30417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:00.581805 30422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:00.581717 30425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:00.581763 30423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:00.582737 30417 server_base.cc:1034] running on GCE node
I20260807 18:28:00.583380 30417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:00.584388 30417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:00.585619 30417 hybrid_clock.cc:648] HybridClock initialized: now 1786127280585567 us; error 62 us; skew 500 ppm
I20260807 18:28:00.585978 30417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:00 dist-test-slave-1xrh krb5kdc[30170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:00.595274 30417 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:28:00.596889 30417 webserver.cc:460] Webserver started at http://127.27.1.194:36101/ using document root <none> and password file <none>
I20260807 18:28:00.597605 30417 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:00.597759 30417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:00.598114 30417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:00.601100 30417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "f05f2230a6fc44cfb7588c19187182f4"
format_stamp: "Formatted at 2026-08-07 18:28:00 on dist-test-slave-1xrh"
server_key: "f1e0b42809a2fa1d6730151345ac1bd4"
server_key_iv: "1aecd1fe7172073d6c7fa3dda9e42c04"
server_key_version: "encryptionkey@0"
I20260807 18:28:00.601951 30417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "f05f2230a6fc44cfb7588c19187182f4"
format_stamp: "Formatted at 2026-08-07 18:28:00 on dist-test-slave-1xrh"
server_key: "f1e0b42809a2fa1d6730151345ac1bd4"
server_key_iv: "1aecd1fe7172073d6c7fa3dda9e42c04"
server_key_version: "encryptionkey@0"
I20260807 18:28:00.608101 30417 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 18:28:00.611748 30432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:00.613164 30417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 18:28:00.613346 30417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "f05f2230a6fc44cfb7588c19187182f4"
format_stamp: "Formatted at 2026-08-07 18:28:00 on dist-test-slave-1xrh"
server_key: "f1e0b42809a2fa1d6730151345ac1bd4"
server_key_iv: "1aecd1fe7172073d6c7fa3dda9e42c04"
server_key_version: "encryptionkey@0"
I20260807 18:28:00.613556 30417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:00.624943 30417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:00.629338 30417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:00.629657 30417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:00.630617 30417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:00.632063 30417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:00.632170 30417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:00.632324 30417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:00.632400 30417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:00.666428 30417 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:39847
I20260807 18:28:00.666522 30549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:39847 every 8 connection(s)
I20260807 18:28:00.667832 30417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:00.675803 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 30417
I20260807 18:28:00.676074 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:28:00.676577 27655 external_mini_cluster.cc:1428] Setting key dbca9e022388d0374d1a3f396f8631fe
Aug 07 18:28:00 dist-test-slave-1xrh krb5kdc[30170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:00.689062 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:00.670327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:32899 (local address 127.27.1.254:44749)
0807 18:28:00.670508 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:00.670512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:00.671612 (+ 1100us) server_negotiation.cc:407] Connection header received
0807 18:28:00.673479 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:00.673485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:00.673582 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:00.673698 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:00.675192 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:00.675960 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:00.677993 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:00.678714 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:00.678846 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:00.683087 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:00.683092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:00.683095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:00.683171 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:00.685923 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:00.686695 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:00.686706 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:00.686709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:00.686838 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:00.687341 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:00.687349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:00.687353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:00.687622 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:00.687780 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:00.688472 (+ 692us) server_negotiation.cc:299] Negotiation successful
0807 18:28:00.688705 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:28:00.690095 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:00.670715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44749 (local address 127.27.1.194:32899)
0807 18:28:00.671455 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:00.671492 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:00.673202 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:00.673934 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:00.673942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:00.674446 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:00.674996 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:00.675014 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:00.676129 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:00.676137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:00.677806 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:00.677825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:00.678962 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:00.678970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:00.679305 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:00.680062 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:00.680096 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:00.682854 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:00.686075 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:00.686084 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:00.686100 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:00.686533 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:00.686954 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:00.686959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:00.686963 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:00.687221 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:00.687753 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:00.687761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:00.688102 (+ 341us) client_negotiation.cc:769] Sending connection context
0807 18:28:00.688369 (+ 267us) client_negotiation.cc:240] Negotiation successful
0807 18:28:00.688907 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":277,"threads_started":1}
I20260807 18:28:00.691732 30550 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44749
I20260807 18:28:00.692051 30550 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:00.692723 30550 heartbeater.cc:511] Master 127.27.1.254:44749 requested a full tablet report, sending...
I20260807 18:28:00.694253 30215 ts_manager.cc:194] Registered new tserver with Master: f05f2230a6fc44cfb7588c19187182f4 (127.27.1.194:39847)
I20260807 18:28:00.695165 30215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:32899
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:28:00.726785 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44749
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:01.070784 30555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:01.071046 30555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:01.071195 30555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:01.078258 30555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:01.078357 30555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:01.078469 30555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:01.088763 30555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:01.090353 30555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:01.092509 30555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:01.107012 30563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:01.107070 30562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:01.106978 30565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:01.108114 30555 server_base.cc:1034] running on GCE node
I20260807 18:28:01.108834 30555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:01.109850 30555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:01.111078 30555 hybrid_clock.cc:648] HybridClock initialized: now 1786127281111025 us; error 68 us; skew 500 ppm
I20260807 18:28:01.111435 30555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:01 dist-test-slave-1xrh krb5kdc[30170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127281, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:01.114671 30555 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:28:01.116190 30555 webserver.cc:460] Webserver started at http://127.27.1.195:36477/ using document root <none> and password file <none>
I20260807 18:28:01.117002 30555 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:01.117195 30555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:01.117544 30555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:01.120710 30555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "7c82d8f479784562968d69be0d406c2a"
format_stamp: "Formatted at 2026-08-07 18:28:01 on dist-test-slave-1xrh"
server_key: "37df1416d3f50e7afcfa37c84c3d3de6"
server_key_iv: "bf2b02e5a0fa16326afb035191270c22"
server_key_version: "encryptionkey@0"
I20260807 18:28:01.121529 30555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "7c82d8f479784562968d69be0d406c2a"
format_stamp: "Formatted at 2026-08-07 18:28:01 on dist-test-slave-1xrh"
server_key: "37df1416d3f50e7afcfa37c84c3d3de6"
server_key_iv: "bf2b02e5a0fa16326afb035191270c22"
server_key_version: "encryptionkey@0"
I20260807 18:28:01.126941 30555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 18:28:01.130388 30572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:01.131796 30555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:01.131978 30555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "7c82d8f479784562968d69be0d406c2a"
format_stamp: "Formatted at 2026-08-07 18:28:01 on dist-test-slave-1xrh"
server_key: "37df1416d3f50e7afcfa37c84c3d3de6"
server_key_iv: "bf2b02e5a0fa16326afb035191270c22"
server_key_version: "encryptionkey@0"
I20260807 18:28:01.132175 30555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:01.155661 30555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:01.160053 30555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:01.160357 30555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:01.161295 30555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:01.162655 30555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:01.162763 30555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:01.162914 30555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:01.162989 30555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:01.192540 30411 heartbeater.cc:503] Master 127.27.1.254:44749 was elected leader, sending a full tablet report...
I20260807 18:28:01.196897 30555 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:37621
I20260807 18:28:01.196955 30688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:37621 every 8 connection(s)
I20260807 18:28:01.198318 30555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:01.206432 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 30555
I20260807 18:28:01.206662 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:28:01.207191 27655 external_mini_cluster.cc:1428] Setting key 1df53e3cf9df2450d6d01de2661717cc
Aug 07 18:28:01 dist-test-slave-1xrh krb5kdc[30170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127281, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:01.218977 30691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:01.200941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:42485 (local address 127.27.1.254:44749)
0807 18:28:01.201420 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:01.201424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:01.202284 (+ 860us) server_negotiation.cc:407] Connection header received
0807 18:28:01.204261 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:01.204267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:01.204375 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:01.204497 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:01.205951 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:01.206862 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:01.208786 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:01.209336 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:01.209471 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:01.213018 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:01.213025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:01.213029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:01.213069 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:01.215798 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:01.216521 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:01.216526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:01.216532 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:01.216659 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:01.217166 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:01.217170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:01.217172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:01.217462 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:01.217642 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:01.218372 (+ 730us) server_negotiation.cc:299] Negotiation successful
0807 18:28:01.218590 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":254,"threads_started":1}
I20260807 18:28:01.219882 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:01.201344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44749 (local address 127.27.1.195:42485)
0807 18:28:01.202129 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:01.202176 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:01.203997 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:01.204725 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:01.204732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:01.205246 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:01.205776 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:01.205794 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:01.207018 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:01.207023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:01.208650 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:01.208663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:01.209575 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:01.209582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:01.209805 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:01.210545 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:01.210566 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:01.212814 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:01.215940 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:01.215948 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:01.215963 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:01.216373 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:01.216774 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:01.216779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:01.216783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:01.217036 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:01.217617 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:01.217625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:01.217949 (+ 324us) client_negotiation.cc:769] Sending connection context
0807 18:28:01.218259 (+ 310us) client_negotiation.cc:240] Negotiation successful
0807 18:28:01.218758 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":304,"threads_started":1}
I20260807 18:28:01.221441 30689 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44749
I20260807 18:28:01.221766 30689 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:01.222687 30689 heartbeater.cc:511] Master 127.27.1.254:44749 requested a full tablet report, sending...
I20260807 18:28:01.224115 30215 ts_manager.cc:194] Registered new tserver with Master: 7c82d8f479784562968d69be0d406c2a (127.27.1.195:37621)
I20260807 18:28:01.224917 30215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:42485
I20260807 18:28:01.234730 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 18:28:01.272238 30691 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:01.269630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.27.1.254:44749)
0807 18:28:01.269819 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:01.269823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:01.270284 (+ 461us) server_negotiation.cc:407] Connection header received
0807 18:28:01.270452 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:01.270456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:01.270524 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:01.270617 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:01.271941 (+ 1324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44634: BlockingRecv error: recv got EOF from 127.0.0.1:44634 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 18:28:01.275400 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30279
W20260807 18:28:01.278302 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30279: Not authorized: Client connection negotiation failed: client connection to 127.27.1.193:41647: server requires authentication, but client does not have Kerberos credentials available
I20260807 18:28:01.278487 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 30279
I20260807 18:28:01.300026 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30417
W20260807 18:28:01.302850 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30417: Not authorized: Client connection negotiation failed: client connection to 127.27.1.194:39847: server requires authentication, but client does not have Kerberos credentials available
I20260807 18:28:01.303042 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 30417
I20260807 18:28:01.325104 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30555
W20260807 18:28:01.328374 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30555: Not authorized: Client connection negotiation failed: client connection to 127.27.1.195:37621: server requires authentication, but client does not have Kerberos credentials available
I20260807 18:28:01.328500 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 30555
I20260807 18:28:01.346601 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30183
I20260807 18:28:01.347883 30247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:01.454610 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 30183
2026-08-07T18:28:01Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3739 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:28:01 dist-test-slave-1xrh krb5kdc[30710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:28:01 dist-test-slave-1xrh krb5kdc[30710](info): set up 2 sockets
Aug 07 18:28:01 dist-test-slave-1xrh krb5kdc[30710](info): commencing operation
krb5kdc: starting...
W20260807 18:28:02.593333 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:28:02 dist-test-slave-1xrh krb5kdc[30710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:28:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:28:02.810451 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38165
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:38165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:03.131806 30723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:03.132108 30723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:03.132305 30723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:03.139532 30723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:03.139631 30723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:03.139741 30723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:03.139839 30723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:03.139940 30723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:03.149693 30723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:38165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:03.151015 30723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:03.152338 30723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:03.164446 30728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:03.164415 30731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:03.164417 30729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:03.165330 30723 server_base.cc:1034] running on GCE node
I20260807 18:28:03.166078 30723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:03.167548 30723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:03.168773 30723 hybrid_clock.cc:648] HybridClock initialized: now 1786127283168733 us; error 35 us; skew 500 ppm
I20260807 18:28:03.169142 30723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:03 dist-test-slave-1xrh krb5kdc[30710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:03.172268 30723 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:28:03.173738 30723 webserver.cc:460] Webserver started at http://127.27.1.254:39579/ using document root <none> and password file <none>
I20260807 18:28:03.174379 30723 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:03.174485 30723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:03.174746 30723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:03.177769 30723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "07987c160ef94c139ea3529993868cb3"
format_stamp: "Formatted at 2026-08-07 18:28:03 on dist-test-slave-1xrh"
server_key: "7bb1697b3546429ad21da0a6bcd11aae"
server_key_iv: "4ceee84468d7c6303971486304823141"
server_key_version: "encryptionkey@0"
I20260807 18:28:03.178536 30723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "07987c160ef94c139ea3529993868cb3"
format_stamp: "Formatted at 2026-08-07 18:28:03 on dist-test-slave-1xrh"
server_key: "7bb1697b3546429ad21da0a6bcd11aae"
server_key_iv: "4ceee84468d7c6303971486304823141"
server_key_version: "encryptionkey@0"
I20260807 18:28:03.183811 30723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260807 18:28:03.187310 30738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:03.188608 30723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:28:03.188798 30723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "07987c160ef94c139ea3529993868cb3"
format_stamp: "Formatted at 2026-08-07 18:28:03 on dist-test-slave-1xrh"
server_key: "7bb1697b3546429ad21da0a6bcd11aae"
server_key_iv: "4ceee84468d7c6303971486304823141"
server_key_version: "encryptionkey@0"
I20260807 18:28:03.189018 30723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:03.205209 30723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:03.209108 30723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:03.209398 30723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:03.227748 30723 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:38165
I20260807 18:28:03.227788 30791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:38165 every 8 connection(s)
I20260807 18:28:03.229136 30723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:03.230266 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 30723
I20260807 18:28:03.230558 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:03.231406 27655 external_mini_cluster.cc:1428] Setting key 519b43511f6c68b0f8378a8c96fb3084
I20260807 18:28:03.234051 30792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 18:28:03 dist-test-slave-1xrh krb5kdc[30710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:03.247787 30792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3: Bootstrap starting.
I20260807 18:28:03.250980 30792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:03.252112 30792 log.cc:824] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:03.255003 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:03.233306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.27.1.254:38165)
0807 18:28:03.234155 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:03.234172 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:03.234222 (+ 50us) server_negotiation.cc:407] Connection header received
0807 18:28:03.236005 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:03.236030 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:03.236403 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:03.236783 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:03.238734 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:03.240166 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:03.241954 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:03.242624 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:03.242751 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:03.247607 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:03.247616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:03.247633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:03.247681 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:03.250767 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:03.251586 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:03.251594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:03.251601 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:03.251766 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:03.252177 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:03.252182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:03.252187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:03.252791 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:03.253130 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:03.253622 (+ 492us) server_negotiation.cc:299] Negotiation successful
0807 18:28:03.254040 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":390,"threads_started":1}
I20260807 18:28:03.255206 30792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3: No bootstrap required, opened a new log
I20260807 18:28:03.258759 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:28:03.262785 30792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07987c160ef94c139ea3529993868cb3" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38165 } }
I20260807 18:28:03.263209 30792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:03.263325 30792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07987c160ef94c139ea3529993868cb3, State: Initialized, Role: FOLLOWER
I20260807 18:28:03.263947 30792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07987c160ef94c139ea3529993868cb3" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38165 } }
I20260807 18:28:03.264204 30792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:03.264356 30792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:03.264546 30792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:03.266748 30792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07987c160ef94c139ea3529993868cb3" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38165 } }
I20260807 18:28:03.267468 30792 leader_election.cc:302] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07987c160ef94c139ea3529993868cb3; no voters:
I20260807 18:28:03.268702 30792 leader_election.cc:288] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:03.268775 30798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:03.271005 30792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:03.275409 30798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 07987c160ef94c139ea3529993868cb3, State: Running, Role: LEADER
I20260807 18:28:03.276098 30798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07987c160ef94c139ea3529993868cb3" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38165 } }
I20260807 18:28:03.282776 30799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07987c160ef94c139ea3529993868cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07987c160ef94c139ea3529993868cb3" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38165 } } }
I20260807 18:28:03.283259 30799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:03.283998 30800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07987c160ef94c139ea3529993868cb3. Latest consensus state: current_term: 1 leader_uuid: "07987c160ef94c139ea3529993868cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07987c160ef94c139ea3529993868cb3" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38165 } } }
I20260807 18:28:03.284444 30800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:03.288476 30804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:03.301862 30804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:03.319641 30804 catalog_manager.cc:1347] Generated new cluster ID: f6505818bb0140adb6dba350128f5435
I20260807 18:28:03.319857 30804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:03.320341 30821 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 18:28:03.335269 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:03.325946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.27.1.254:38165)
0807 18:28:03.326139 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:03.326148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:03.326743 (+ 595us) server_negotiation.cc:407] Connection header received
0807 18:28:03.326922 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:03.326928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:03.327009 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:03.327114 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:03.328497 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:03.329610 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:03.331601 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:03.332232 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:03.332345 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:03.333607 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:03.333614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:03.333618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:03.334410 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:03.334481 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:03.334521 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:03.334629 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:03.334797 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 18:28:03.334888 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:28:03.363567 30804 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:03.364912 30804 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:03.373348 30804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07987c160ef94c139ea3529993868cb3: Generated new TSK 0
I20260807 18:28:03.374173 30804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:28:03.388792 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30723
I20260807 18:28:03.389947 30787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:03.483245 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 30723
2026-08-07T18:28:03Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2022 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:28:03 dist-test-slave-1xrh krb5kdc[30829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:28:03 dist-test-slave-1xrh krb5kdc[30829](info): set up 2 sockets
Aug 07 18:28:03 dist-test-slave-1xrh krb5kdc[30829](info): commencing operation
krb5kdc: starting...
W20260807 18:28:04.605489 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:28:04 dist-test-slave-1xrh krb5kdc[30829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:28:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:28:04.818431 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:43423
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:43423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:05.143633 30842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:05.143944 30842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:05.144142 30842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:05.151818 30842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:05.151916 30842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:05.152040 30842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:05.152132 30842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:05.152215 30842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:05.161862 30842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:43423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:43423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:05.163206 30842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:05.164518 30842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:05.176416 30848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:05.176384 30849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:05.176590 30851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:05.177222 30842 server_base.cc:1034] running on GCE node
I20260807 18:28:05.177980 30842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:05.179323 30842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:05.180528 30842 hybrid_clock.cc:648] HybridClock initialized: now 1786127285180472 us; error 53 us; skew 500 ppm
I20260807 18:28:05.180855 30842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:05 dist-test-slave-1xrh krb5kdc[30829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:05.183925 30842 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:28:05.185225 30842 webserver.cc:460] Webserver started at http://127.27.1.254:33363/ using document root <none> and password file <none>
I20260807 18:28:05.185840 30842 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:05.185938 30842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:05.186269 30842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:05.189158 30842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe"
format_stamp: "Formatted at 2026-08-07 18:28:05 on dist-test-slave-1xrh"
server_key: "3c8f9391f98396c91c8bd53120272f45"
server_key_iv: "a51a39dac02ca8cec0b2935db5b6a4a4"
server_key_version: "encryptionkey@0"
I20260807 18:28:05.189862 30842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe"
format_stamp: "Formatted at 2026-08-07 18:28:05 on dist-test-slave-1xrh"
server_key: "3c8f9391f98396c91c8bd53120272f45"
server_key_iv: "a51a39dac02ca8cec0b2935db5b6a4a4"
server_key_version: "encryptionkey@0"
I20260807 18:28:05.194865 30842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260807 18:28:05.198154 30858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:05.199667 30842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:05.199894 30842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe"
format_stamp: "Formatted at 2026-08-07 18:28:05 on dist-test-slave-1xrh"
server_key: "3c8f9391f98396c91c8bd53120272f45"
server_key_iv: "a51a39dac02ca8cec0b2935db5b6a4a4"
server_key_version: "encryptionkey@0"
I20260807 18:28:05.200109 30842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:05.212899 30842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:05.217036 30842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:05.217343 30842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:05.237551 30842 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:43423
I20260807 18:28:05.237650 30910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:43423 every 8 connection(s)
I20260807 18:28:05.238976 30842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:05.243028 30911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:05.247539 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 30842
I20260807 18:28:05.247771 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:05.248617 27655 external_mini_cluster.cc:1428] Setting key 16a5b9bbd3a9bce336a1ff1b0a0d056f
I20260807 18:28:05.255168 30911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe: Bootstrap starting.
I20260807 18:28:05.258797 30911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:05.259974 30911 log.cc:824] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:05.263628 30911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe: No bootstrap required, opened a new log
Aug 07 18:28:05 dist-test-slave-1xrh krb5kdc[30829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:05.271359 30911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43423 } }
I20260807 18:28:05.271809 30911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:05.271925 30911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b69fdd51a1b404a81ebb0aa47c46cbe, State: Initialized, Role: FOLLOWER
I20260807 18:28:05.272514 30911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43423 } }
I20260807 18:28:05.272751 30911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:05.272897 30915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:05.250724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.27.1.254:43423)
0807 18:28:05.251567 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:05.251580 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:05.251624 (+ 44us) server_negotiation.cc:407] Connection header received
0807 18:28:05.253628 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:05.253651 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:05.254014 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:05.254331 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:05.256189 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:05.257547 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:05.259394 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:05.260100 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:05.260310 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:05.265224 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:05.265235 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:05.265248 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:05.265293 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:05.268548 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:05.269284 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:05.269291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:05.269297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:05.269469 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:05.269880 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:05.269887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:05.269889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:05.270408 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:05.270741 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:05.271140 (+ 399us) server_negotiation.cc:299] Negotiation successful
0807 18:28:05.271576 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":378,"threads_started":1}
I20260807 18:28:05.272912 30911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:05.273142 30911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:05.275722 30911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43423 } }
I20260807 18:28:05.276233 30911 leader_election.cc:302] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b69fdd51a1b404a81ebb0aa47c46cbe; no voters:
I20260807 18:28:05.276783 30911 leader_election.cc:288] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:05.276927 30917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:05.277736 30917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [term 1 LEADER]: Becoming Leader. State: Replica: 1b69fdd51a1b404a81ebb0aa47c46cbe, State: Running, Role: LEADER
I20260807 18:28:05.278359 30917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43423 } }
I20260807 18:28:05.278860 30911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:05.281216 30919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b69fdd51a1b404a81ebb0aa47c46cbe. Latest consensus state: current_term: 1 leader_uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43423 } } }
I20260807 18:28:05.281646 30919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:05.282104 30918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fdd51a1b404a81ebb0aa47c46cbe" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43423 } } }
I20260807 18:28:05.282516 30918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:05.283895 30926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:05.291657 30926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:05.305271 30926 catalog_manager.cc:1347] Generated new cluster ID: dcf6570a5e3f4674b1569fe42c53578e
I20260807 18:28:05.305425 30926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:05.323376 30926 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:05.325449 30926 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:05.334136 30926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b69fdd51a1b404a81ebb0aa47c46cbe: Generated new TSK 0
I20260807 18:28:05.335238 30926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:28:05.392850 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:43423
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:05.729393 30938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:05.729669 30938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:05.729765 30938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:05.737036 30938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:05.737144 30938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:05.737257 30938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:05.746933 30938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:43423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:05.748255 30938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:05.749562 30938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:05.762719 30944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:05.762625 30946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:05.763175 30943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:05.763530 30938 server_base.cc:1034] running on GCE node
I20260807 18:28:05.764209 30938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:05.765202 30938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:05.766439 30938 hybrid_clock.cc:648] HybridClock initialized: now 1786127285766399 us; error 56 us; skew 500 ppm
I20260807 18:28:05.766767 30938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:05 dist-test-slave-1xrh krb5kdc[30829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:05.776996 30938 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:28:05.778730 30938 webserver.cc:460] Webserver started at http://127.27.1.193:32831/ using document root <none> and password file <none>
I20260807 18:28:05.779585 30938 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:05.779759 30938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:05.780265 30938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:05.783964 30938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "a0672fbe5c1b450db503e12742849e8d"
format_stamp: "Formatted at 2026-08-07 18:28:05 on dist-test-slave-1xrh"
server_key: "8e46c5562d6858923789c8cdd6f1e3e2"
server_key_iv: "4b8ecd943e5579484f61dec7d40b87f3"
server_key_version: "encryptionkey@0"
I20260807 18:28:05.784799 30938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "a0672fbe5c1b450db503e12742849e8d"
format_stamp: "Formatted at 2026-08-07 18:28:05 on dist-test-slave-1xrh"
server_key: "8e46c5562d6858923789c8cdd6f1e3e2"
server_key_iv: "4b8ecd943e5579484f61dec7d40b87f3"
server_key_version: "encryptionkey@0"
I20260807 18:28:05.790186 30938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 18:28:05.793879 30953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:05.795347 30938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:05.795528 30938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "a0672fbe5c1b450db503e12742849e8d"
format_stamp: "Formatted at 2026-08-07 18:28:05 on dist-test-slave-1xrh"
server_key: "8e46c5562d6858923789c8cdd6f1e3e2"
server_key_iv: "4b8ecd943e5579484f61dec7d40b87f3"
server_key_version: "encryptionkey@0"
I20260807 18:28:05.795734 30938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:05.816998 30938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:05.821650 30938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:05.821964 30938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:05.822966 30938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:05.824436 30938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:05.824541 30938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:05.824694 30938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:05.824769 30938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:05.859669 30938 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:43545
I20260807 18:28:05.859760 31072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:43545 every 8 connection(s)
I20260807 18:28:05.861021 30938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:05.864975 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 30938
I20260807 18:28:05.865211 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:28:05.866070 27655 external_mini_cluster.cc:1428] Setting key a46cef7c074272b81da3e2e7fcdbc9c8
Aug 07 18:28:05 dist-test-slave-1xrh krb5kdc[30829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:05.885171 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:05.863620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:45933 (local address 127.27.1.254:43423)
0807 18:28:05.864080 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:05.864085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:05.865048 (+ 963us) server_negotiation.cc:407] Connection header received
0807 18:28:05.867463 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:05.867469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:05.867547 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:05.867665 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:05.869494 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:05.870185 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:05.872675 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:05.873067 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:05.873168 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:05.877747 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:05.877756 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:05.877764 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:05.877809 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:05.881562 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:05.882260 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:05.882268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:05.882275 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:05.882430 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:05.882937 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:05.882945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:05.882951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:05.883377 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:05.883534 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:05.884402 (+ 868us) server_negotiation.cc:299] Negotiation successful
0807 18:28:05.884666 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":256,"threads_started":1}
I20260807 18:28:05.885686 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:05.864040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:43423 (local address 127.27.1.193:45933)
0807 18:28:05.864863 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:05.864935 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:05.867205 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:05.867861 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:05.867900 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:05.868534 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:05.869313 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:05.869342 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:05.870357 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:05.870366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:05.872525 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:05.872543 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:05.873300 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:05.873310 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:05.873640 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:05.874391 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:05.874434 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:05.877446 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:05.881702 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:05.881709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:05.881721 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:05.882124 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:05.882556 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:05.882559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:05.882562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:05.882785 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:05.883518 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:05.883524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:05.883973 (+ 449us) client_negotiation.cc:769] Sending connection context
0807 18:28:05.884207 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 18:28:05.884572 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":278,"threads_started":1}
I20260807 18:28:05.887363 31073 heartbeater.cc:347] Connected to a master server at 127.27.1.254:43423
I20260807 18:28:05.887691 31073 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:05.888435 31073 heartbeater.cc:511] Master 127.27.1.254:43423 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
I20260807 18:28:05.890606 30875 ts_manager.cc:194] Registered new tserver with Master: a0672fbe5c1b450db503e12742849e8d (127.27.1.193:43545)
I20260807 18:28:05.892292 30875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:45933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:28:05.918481 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:43423
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:06.253386 31079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:06.253726 31079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:06.253886 31079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:06.264149 31079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:06.264256 31079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:06.264369 31079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:06.273890 31079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:43423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:06.275059 31079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:06.276386 31079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:06.289997 31085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:06.290153 31088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:06.290175 31086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:06.290741 31079 server_base.cc:1034] running on GCE node
I20260807 18:28:06.291414 31079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:06.292378 31079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:06.293612 31079 hybrid_clock.cc:648] HybridClock initialized: now 1786127286293572 us; error 30 us; skew 500 ppm
I20260807 18:28:06.293928 31079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:06 dist-test-slave-1xrh krb5kdc[30829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:06.297200 31079 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:28:06.298840 31079 webserver.cc:460] Webserver started at http://127.27.1.194:41429/ using document root <none> and password file <none>
I20260807 18:28:06.299733 31079 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:06.299882 31079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:06.300366 31079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:06.303597 31079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "dde84963efda49ffbd6d5c741dca1a2c"
format_stamp: "Formatted at 2026-08-07 18:28:06 on dist-test-slave-1xrh"
server_key: "918c35d923d61ff7eb29b1861a9da553"
server_key_iv: "1386aad11b2dd9cf8e6bbec835bbff10"
server_key_version: "encryptionkey@0"
I20260807 18:28:06.304525 31079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "dde84963efda49ffbd6d5c741dca1a2c"
format_stamp: "Formatted at 2026-08-07 18:28:06 on dist-test-slave-1xrh"
server_key: "918c35d923d61ff7eb29b1861a9da553"
server_key_iv: "1386aad11b2dd9cf8e6bbec835bbff10"
server_key_version: "encryptionkey@0"
I20260807 18:28:06.310087 31079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:28:06.313723 31095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:06.315217 31079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:06.315433 31079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "dde84963efda49ffbd6d5c741dca1a2c"
format_stamp: "Formatted at 2026-08-07 18:28:06 on dist-test-slave-1xrh"
server_key: "918c35d923d61ff7eb29b1861a9da553"
server_key_iv: "1386aad11b2dd9cf8e6bbec835bbff10"
server_key_version: "encryptionkey@0"
I20260807 18:28:06.315650 31079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:06.339978 31079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:06.344231 31079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:06.344565 31079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:06.345599 31079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:06.346987 31079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:06.347092 31079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:06.347263 31079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:06.347339 31079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:06.381649 31079 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:36795
I20260807 18:28:06.381695 31212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:36795 every 8 connection(s)
I20260807 18:28:06.383051 31079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:06.389741 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 31079
I20260807 18:28:06.389917 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:28:06.390504 27655 external_mini_cluster.cc:1428] Setting key bba61ff309fc35ddc1039bac30b78f79
Aug 07 18:28:06 dist-test-slave-1xrh krb5kdc[30829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:06.404940 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:06.385753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:52649 (local address 127.27.1.254:43423)
0807 18:28:06.386266 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:06.386272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:06.387115 (+ 843us) server_negotiation.cc:407] Connection header received
0807 18:28:06.388907 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:06.388913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:06.389007 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:06.389138 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:06.391158 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.391998 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.394269 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.394696 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.394793 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:06.398955 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:06.398960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:06.398964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:06.399004 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:06.401770 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:06.402524 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:06.402530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:06.402535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:06.402679 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:06.403118 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:06.403147 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:06.403150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:06.403459 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:06.403633 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:06.404317 (+ 684us) server_negotiation.cc:299] Negotiation successful
0807 18:28:06.404539 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":268,"threads_started":1}
I20260807 18:28:06.405920 31216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:06.386226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:43423 (local address 127.27.1.194:52649)
0807 18:28:06.386959 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:06.387003 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:06.388640 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:06.389381 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:06.389394 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:06.390135 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:06.390935 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:06.390961 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.392167 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.392173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:06.394087 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:06.394106 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.394936 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.394945 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:06.395304 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:06.396039 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:06.396072 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:06.398732 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:06.401927 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:06.401936 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:06.401953 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:06.402381 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:06.402793 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:06.402797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:06.402801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:06.403006 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:06.403610 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:06.403618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:06.403962 (+ 344us) client_negotiation.cc:769] Sending connection context
0807 18:28:06.404225 (+ 263us) client_negotiation.cc:240] Negotiation successful
0807 18:28:06.404846 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":262,"threads_started":1}
I20260807 18:28:06.407410 31213 heartbeater.cc:347] Connected to a master server at 127.27.1.254:43423
I20260807 18:28:06.407737 31213 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:06.408597 31213 heartbeater.cc:511] Master 127.27.1.254:43423 requested a full tablet report, sending...
I20260807 18:28:06.410184 30875 ts_manager.cc:194] Registered new tserver with Master: dde84963efda49ffbd6d5c741dca1a2c (127.27.1.194:36795)
I20260807 18:28:06.410991 30875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:52649
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:28:06.442148 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:43423
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:06.770263 31219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:06.770547 31219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:06.770712 31219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:06.779350 31219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:06.779474 31219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:06.779656 31219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:06.789614 31219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:43423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:06.790918 31219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:06.792371 31219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:06.805595 31224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:06.805576 31225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:06.805644 31227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:06.806974 31219 server_base.cc:1034] running on GCE node
I20260807 18:28:06.807607 31219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:06.808524 31219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:06.809721 31219 hybrid_clock.cc:648] HybridClock initialized: now 1786127286809660 us; error 50 us; skew 500 ppm
I20260807 18:28:06.810001 31219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:06 dist-test-slave-1xrh krb5kdc[30829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:06.813127 31219 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:28:06.814543 31219 webserver.cc:460] Webserver started at http://127.27.1.195:42885/ using document root <none> and password file <none>
I20260807 18:28:06.815299 31219 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:06.815451 31219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:06.815766 31219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:06.818598 31219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "5e01103c686747979d853b738ce95df1"
format_stamp: "Formatted at 2026-08-07 18:28:06 on dist-test-slave-1xrh"
server_key: "d3ee7319a3d80ac6556e536699d3aded"
server_key_iv: "8a9c586f530b2cf3e770dbac40bfd8fc"
server_key_version: "encryptionkey@0"
I20260807 18:28:06.819541 31219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "5e01103c686747979d853b738ce95df1"
format_stamp: "Formatted at 2026-08-07 18:28:06 on dist-test-slave-1xrh"
server_key: "d3ee7319a3d80ac6556e536699d3aded"
server_key_iv: "8a9c586f530b2cf3e770dbac40bfd8fc"
server_key_version: "encryptionkey@0"
I20260807 18:28:06.824756 31219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:28:06.828151 31234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:06.829541 31219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:28:06.829715 31219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "5e01103c686747979d853b738ce95df1"
format_stamp: "Formatted at 2026-08-07 18:28:06 on dist-test-slave-1xrh"
server_key: "d3ee7319a3d80ac6556e536699d3aded"
server_key_iv: "8a9c586f530b2cf3e770dbac40bfd8fc"
server_key_version: "encryptionkey@0"
I20260807 18:28:06.829928 31219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:06.841689 31219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:06.845387 31219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:06.845669 31219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:06.846558 31219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:06.847970 31219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:06.848073 31219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:06.848223 31219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:06.848308 31219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:06.882416 31219 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:46331
I20260807 18:28:06.882455 31355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:46331 every 8 connection(s)
I20260807 18:28:06.883769 31219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:06.893167 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 31219
I20260807 18:28:06.893352 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:28:06.893808 27655 external_mini_cluster.cc:1428] Setting key f9c4593389f220ec7f44794cb3f987c7
I20260807 18:28:06.895938 31073 heartbeater.cc:503] Master 127.27.1.254:43423 was elected leader, sending a full tablet report...
Aug 07 18:28:06 dist-test-slave-1xrh krb5kdc[30829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:06.903232 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:06.886418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:52037 (local address 127.27.1.254:43423)
0807 18:28:06.886610 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:06.886614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:06.887722 (+ 1108us) server_negotiation.cc:407] Connection header received
0807 18:28:06.889464 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:06.889469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:06.889577 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:06.889692 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:06.891105 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.891905 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.893431 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.893824 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.893961 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:06.897894 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:06.897899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:06.897902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:06.897932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:06.900685 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:06.901344 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:06.901347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:06.901350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:06.901465 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:06.901824 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:06.901827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:06.901829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:06.902083 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:06.902205 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:06.902790 (+ 585us) server_negotiation.cc:299] Negotiation successful
0807 18:28:06.902947 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 18:28:06.904330 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:06.886814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:43423 (local address 127.27.1.195:52037)
0807 18:28:06.887562 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:06.887604 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:06.889195 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:06.889916 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:06.889923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:06.890393 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:06.890935 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:06.890951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.892059 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.892066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:06.893261 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:06.893273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.894047 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.894055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:06.894324 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:06.895116 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:06.895169 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:06.897719 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:06.900804 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:06.900810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:06.900821 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:06.901242 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:06.901560 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:06.901563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:06.901565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:06.901731 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:06.902197 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:06.902202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:06.902484 (+ 282us) client_negotiation.cc:769] Sending connection context
0807 18:28:06.902707 (+ 223us) client_negotiation.cc:240] Negotiation successful
0807 18:28:06.903073 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":324,"threads_started":1}
I20260807 18:28:06.905642 31356 heartbeater.cc:347] Connected to a master server at 127.27.1.254:43423
I20260807 18:28:06.905900 31356 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:06.906551 31356 heartbeater.cc:511] Master 127.27.1.254:43423 requested a full tablet report, sending...
I20260807 18:28:06.908218 30875 ts_manager.cc:194] Registered new tserver with Master: 5e01103c686747979d853b738ce95df1 (127.27.1.195:46331)
I20260807 18:28:06.909336 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:28:06.909332 30875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:52037
Aug 07 18:28:06 dist-test-slave-1xrh krb5kdc[30829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127286, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 18:28:06 dist-test-slave-1xrh krb5kdc[30829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127286, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:06.963418 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:06.949142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.27.1.254:43423)
0807 18:28:06.949420 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:06.949430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:06.949813 (+ 383us) server_negotiation.cc:407] Connection header received
0807 18:28:06.950003 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:06.950007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:06.950083 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:06.950173 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:06.951768 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.952629 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.953849 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.954274 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.954390 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:06.958331 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:06.958336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:06.958339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:06.958375 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:06.960978 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:06.961708 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:06.961712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:06.961714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:06.961833 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:06.962153 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:06.962156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:06.962158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:06.962458 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:06.962669 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:06.962920 (+ 251us) server_negotiation.cc:299] Negotiation successful
0807 18:28:06.963076 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 18:28:06.964000 30875 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40042
I20260807 18:28:06.965904 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30938
Aug 07 18:28:06 dist-test-slave-1xrh krb5kdc[30829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127286, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:28:06.980949 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:06.967027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.27.1.193:43545)
0807 18:28:06.967696 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:06.967702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:06.967726 (+ 24us) server_negotiation.cc:407] Connection header received
0807 18:28:06.967782 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:06.967788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:06.967996 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:06.968121 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:06.969146 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.970153 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.971314 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:06.971853 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:06.971951 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:06.975667 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:06.975678 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:06.975691 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:06.975723 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:06.978341 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:06.978927 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:06.978934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:06.978937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:06.979070 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:06.979480 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:06.979485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:06.979487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:06.979888 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:06.980142 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:06.980431 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 18:28:06.980665 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":354,"threads_started":1}
W20260807 18:28:06.982290 31068 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60184
W20260807 18:28:06.983485 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30938: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:28:06.983726 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 30938
I20260807 18:28:07.004856 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 31079
Aug 07 18:28:07 dist-test-slave-1xrh krb5kdc[30829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127286, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:28:07.021252 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:07.005852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.27.1.194:36795)
0807 18:28:07.006452 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:07.006460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:07.006487 (+ 27us) server_negotiation.cc:407] Connection header received
0807 18:28:07.006552 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:07.006558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:07.006767 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:07.006923 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:07.008246 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:07.009500 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:07.010707 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:07.011547 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:07.011653 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:07.015724 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:07.015734 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:07.015749 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:07.015782 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:07.018473 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:07.019061 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:07.019067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:07.019070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:07.019344 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:07.019741 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:07.019745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:07.019747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:07.020107 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:07.020348 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:07.020757 (+ 409us) server_negotiation.cc:299] Negotiation successful
0807 18:28:07.020962 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":308,"threads_started":1}
W20260807 18:28:07.022518 31208 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49278
W20260807 18:28:07.023658 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 31079: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:28:07.023802 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 31079
I20260807 18:28:07.043097 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 31219
Aug 07 18:28:07 dist-test-slave-1xrh krb5kdc[30829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127286, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:28:07.059262 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:07.044104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.27.1.195:46331)
0807 18:28:07.044778 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:07.044784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:07.044814 (+ 30us) server_negotiation.cc:407] Connection header received
0807 18:28:07.044876 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:07.044883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:07.045087 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:07.045214 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:07.046496 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:07.047634 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:07.048880 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:07.049425 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:07.049519 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:07.053724 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:07.053736 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:07.053750 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:07.053783 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:07.056778 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:07.057410 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:07.057416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:07.057418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:07.057538 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:07.057842 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:07.057845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:07.057847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:07.058161 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:07.058414 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:07.058591 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 18:28:07.058778 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":345,"threads_started":1}
W20260807 18:28:07.060549 31351 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43424
W20260807 18:28:07.061596 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 31219: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:28:07.061754 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 31219
I20260807 18:28:07.081169 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 30842
I20260807 18:28:07.082305 30906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:07.185508 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 30842
2026-08-07T18:28:07Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3707 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:28:07 dist-test-slave-1xrh krb5kdc[31373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:28:07 dist-test-slave-1xrh krb5kdc[31373](info): set up 2 sockets
Aug 07 18:28:07 dist-test-slave-1xrh krb5kdc[31373](info): commencing operation
krb5kdc: starting...
W20260807 18:28:08.401544 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:28:08 dist-test-slave-1xrh krb5kdc[31373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:28:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:28:08.624686 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:46787
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:46787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:08.955116 31387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:08.955466 31387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:08.955623 31387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:08.962848 31387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:08.962952 31387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:08.963052 31387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:08.963161 31387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:08.963248 31387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:08.973167 31387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:46787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:46787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:08.974562 31387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:08.975916 31387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:08.987790 31395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:08.987874 31392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:08.987833 31393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:08.988598 31387 server_base.cc:1034] running on GCE node
I20260807 18:28:08.989408 31387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:08.990811 31387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:08.992027 31387 hybrid_clock.cc:648] HybridClock initialized: now 1786127288991976 us; error 30 us; skew 500 ppm
I20260807 18:28:08.992332 31387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:08 dist-test-slave-1xrh krb5kdc[31373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:08.995625 31387 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:28:08.997191 31387 webserver.cc:460] Webserver started at http://127.27.1.254:36475/ using document root <none> and password file <none>
I20260807 18:28:08.997889 31387 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:08.998039 31387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:08.998370 31387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:09.001371 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "619356b8b4f74288b3e3eba0afb05970"
format_stamp: "Formatted at 2026-08-07 18:28:08 on dist-test-slave-1xrh"
server_key: "3066b91496901e0ac7e6ef0d9eb94ed7"
server_key_iv: "db78aa98900624538f91ace2c9105a23"
server_key_version: "encryptionkey@0"
I20260807 18:28:09.002192 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "619356b8b4f74288b3e3eba0afb05970"
format_stamp: "Formatted at 2026-08-07 18:28:08 on dist-test-slave-1xrh"
server_key: "3066b91496901e0ac7e6ef0d9eb94ed7"
server_key_iv: "db78aa98900624538f91ace2c9105a23"
server_key_version: "encryptionkey@0"
I20260807 18:28:09.007614 31387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 18:28:09.010987 31402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:09.012349 31387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:28:09.012552 31387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "619356b8b4f74288b3e3eba0afb05970"
format_stamp: "Formatted at 2026-08-07 18:28:08 on dist-test-slave-1xrh"
server_key: "3066b91496901e0ac7e6ef0d9eb94ed7"
server_key_iv: "db78aa98900624538f91ace2c9105a23"
server_key_version: "encryptionkey@0"
I20260807 18:28:09.012753 31387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:09.031894 31387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:09.036538 31387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:09.036849 31387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:09.055440 31387 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:46787
I20260807 18:28:09.055465 31456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:46787 every 8 connection(s)
I20260807 18:28:09.056805 31387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:09.060443 31457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:09.065759 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 31387
I20260807 18:28:09.065990 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:09.066817 27655 external_mini_cluster.cc:1428] Setting key 1a4c933ebcba3420edccc527b49364fd
I20260807 18:28:09.072091 31457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970: Bootstrap starting.
I20260807 18:28:09.075016 31457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:09.076026 31457 log.cc:824] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:09.078871 31457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970: No bootstrap required, opened a new log
Aug 07 18:28:09 dist-test-slave-1xrh krb5kdc[31373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:09.089419 31457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619356b8b4f74288b3e3eba0afb05970" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46787 } }
I20260807 18:28:09.089876 31457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:09.090013 31457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619356b8b4f74288b3e3eba0afb05970, State: Initialized, Role: FOLLOWER
I20260807 18:28:09.090276 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:09.069038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.27.1.254:46787)
0807 18:28:09.069848 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:09.069859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:09.069900 (+ 41us) server_negotiation.cc:407] Connection header received
0807 18:28:09.071630 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:09.071653 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:09.072056 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:09.072472 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:09.073966 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:09.075258 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:09.076577 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:09.077375 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:09.077569 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:09.081683 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:09.081693 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:09.081711 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:09.081764 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:09.085811 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:09.086444 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:09.086453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:09.086461 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:09.086709 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:09.087062 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:09.087069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:09.087077 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:09.087836 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:09.088128 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:09.088501 (+ 373us) server_negotiation.cc:299] Negotiation successful
0807 18:28:09.088915 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":370,"threads_started":1}
I20260807 18:28:09.090680 31457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619356b8b4f74288b3e3eba0afb05970" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46787 } }
I20260807 18:28:09.091006 31457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:09.091187 31457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:09.091369 31457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:09.093865 31457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619356b8b4f74288b3e3eba0afb05970" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46787 } }
I20260807 18:28:09.094292 31457 leader_election.cc:302] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 619356b8b4f74288b3e3eba0afb05970; no voters:
I20260807 18:28:09.094933 31457 leader_election.cc:288] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:09.095053 31462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:09.097018 31457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:09.097573 31462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [term 1 LEADER]: Becoming Leader. State: Replica: 619356b8b4f74288b3e3eba0afb05970, State: Running, Role: LEADER
I20260807 18:28:09.098084 31462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619356b8b4f74288b3e3eba0afb05970" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46787 } }
I20260807 18:28:09.101686 31463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "619356b8b4f74288b3e3eba0afb05970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619356b8b4f74288b3e3eba0afb05970" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46787 } } }
I20260807 18:28:09.101790 31464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 619356b8b4f74288b3e3eba0afb05970. Latest consensus state: current_term: 1 leader_uuid: "619356b8b4f74288b3e3eba0afb05970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619356b8b4f74288b3e3eba0afb05970" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46787 } } }
I20260807 18:28:09.102067 31463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:09.102151 31464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:09.103662 31472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:09.108306 31472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:09.116535 31472 catalog_manager.cc:1347] Generated new cluster ID: 7ebb3d8dba7841e598d5691e766a59c5
I20260807 18:28:09.116712 31472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:09.137713 31472 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:09.138999 31472 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:09.147701 31472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 619356b8b4f74288b3e3eba0afb05970: Generated new TSK 0
I20260807 18:28:09.148475 31472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:28:09.206689 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46787
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:09.545792 31485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:09.546073 31485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:09.546227 31485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:09.553427 31485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:09.553561 31485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:09.553774 31485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:09.563459 31485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:09.564805 31485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:09.566179 31485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:09.579222 31490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:09.579248 31493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:09.579269 31491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:09.580179 31485 server_base.cc:1034] running on GCE node
I20260807 18:28:09.580762 31485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:09.581689 31485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:09.582952 31485 hybrid_clock.cc:648] HybridClock initialized: now 1786127289582880 us; error 61 us; skew 500 ppm
I20260807 18:28:09.583353 31485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:09 dist-test-slave-1xrh krb5kdc[31373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:09.586725 31485 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:28:09.588233 31485 webserver.cc:460] Webserver started at http://127.27.1.193:44773/ using document root <none> and password file <none>
I20260807 18:28:09.589066 31485 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:09.589213 31485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:09.589510 31485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:09.592473 31485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "c3c15ded5c17479fbc934baf55fd52db"
format_stamp: "Formatted at 2026-08-07 18:28:09 on dist-test-slave-1xrh"
server_key: "b1fb2758abe6f3a5768e34527235a4ba"
server_key_iv: "064a9600b450d3a5e9f11d71c48d2dc9"
server_key_version: "encryptionkey@0"
I20260807 18:28:09.593245 31485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "c3c15ded5c17479fbc934baf55fd52db"
format_stamp: "Formatted at 2026-08-07 18:28:09 on dist-test-slave-1xrh"
server_key: "b1fb2758abe6f3a5768e34527235a4ba"
server_key_iv: "064a9600b450d3a5e9f11d71c48d2dc9"
server_key_version: "encryptionkey@0"
I20260807 18:28:09.598958 31485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 18:28:09.603288 31501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:09.604844 31485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:28:09.605022 31485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "c3c15ded5c17479fbc934baf55fd52db"
format_stamp: "Formatted at 2026-08-07 18:28:09 on dist-test-slave-1xrh"
server_key: "b1fb2758abe6f3a5768e34527235a4ba"
server_key_iv: "064a9600b450d3a5e9f11d71c48d2dc9"
server_key_version: "encryptionkey@0"
I20260807 18:28:09.605232 31485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:09.622912 31485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:09.627336 31485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:09.627614 31485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:09.628552 31485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:09.629873 31485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:09.629943 31485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:09.630105 31485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:09.630156 31485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:09.662777 31485 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:34125
I20260807 18:28:09.662817 31618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:34125 every 8 connection(s)
I20260807 18:28:09.664249 31485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:09.665573 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 31485
I20260807 18:28:09.665794 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:28:09.666429 27655 external_mini_cluster.cc:1428] Setting key 9bd10d7281ccd98f5ca41e78581f8e90
Aug 07 18:28:09 dist-test-slave-1xrh krb5kdc[31373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:09.688925 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:09.666954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:50137 (local address 127.27.1.254:46787)
0807 18:28:09.667649 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:09.667654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:09.668508 (+ 854us) server_negotiation.cc:407] Connection header received
0807 18:28:09.671310 (+ 2802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:09.671314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:09.671379 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:09.671477 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:09.673530 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:09.674248 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:09.676721 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:09.677122 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:09.677261 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:09.681330 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:09.681339 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:09.681347 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:09.681393 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:09.685185 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:09.685987 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:09.685996 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:09.686002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:09.686157 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:09.686651 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:09.686659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:09.686665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:09.687078 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:09.687271 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:09.688126 (+ 855us) server_negotiation.cc:299] Negotiation successful
0807 18:28:09.688414 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":260,"threads_started":1}
I20260807 18:28:09.689613 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:09.667556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:46787 (local address 127.27.1.193:50137)
0807 18:28:09.668342 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:09.668402 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:09.670996 (+ 2594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:09.671740 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:09.671754 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:09.672528 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:09.673355 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:09.673384 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:09.674415 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:09.674423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:09.676579 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:09.676599 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:09.677396 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:09.677402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:09.677616 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:09.678416 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:09.678439 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:09.681080 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:09.685345 (+ 4265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:09.685352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:09.685366 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:09.685833 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:09.686291 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:09.686295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:09.686298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:09.686509 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:09.687240 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:09.687247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:09.687685 (+ 438us) client_negotiation.cc:769] Sending connection context
0807 18:28:09.687937 (+ 252us) client_negotiation.cc:240] Negotiation successful
0807 18:28:09.688354 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":301,"threads_started":1}
I20260807 18:28:09.691437 31619 heartbeater.cc:347] Connected to a master server at 127.27.1.254:46787
I20260807 18:28:09.691780 31619 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
I20260807 18:28:09.692623 31619 heartbeater.cc:511] Master 127.27.1.254:46787 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
I20260807 18:28:09.694947 31420 ts_manager.cc:194] Registered new tserver with Master: c3c15ded5c17479fbc934baf55fd52db (127.27.1.193:34125)
I20260807 18:28:09.697257 31420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:50137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:28:09.722465 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46787
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:10.050362 31626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:10.050637 31626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:10.050730 31626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:10.057940 31626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:10.058041 31626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:10.058239 31626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:10.067548 31626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:10.068713 31626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:10.070039 31626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:10.083020 31631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:10.082983 31632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:10.083055 31635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:10.083931 31626 server_base.cc:1034] running on GCE node
I20260807 18:28:10.084504 31626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:10.085414 31626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:10.086642 31626 hybrid_clock.cc:648] HybridClock initialized: now 1786127290086603 us; error 56 us; skew 500 ppm
I20260807 18:28:10.086948 31626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:10 dist-test-slave-1xrh krb5kdc[31373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:10.090065 31626 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:28:10.091432 31626 webserver.cc:460] Webserver started at http://127.27.1.194:45685/ using document root <none> and password file <none>
I20260807 18:28:10.092054 31626 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:10.092157 31626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:10.092410 31626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:10.095268 31626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "c4fcd8b06ef6451b9f3c2b49e615eda9"
format_stamp: "Formatted at 2026-08-07 18:28:10 on dist-test-slave-1xrh"
server_key: "c0d46efa080c82f3a413b6902b1ff0ae"
server_key_iv: "116eec38e73dfa052d3022f5aa6e7f7a"
server_key_version: "encryptionkey@0"
I20260807 18:28:10.095997 31626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "c4fcd8b06ef6451b9f3c2b49e615eda9"
format_stamp: "Formatted at 2026-08-07 18:28:10 on dist-test-slave-1xrh"
server_key: "c0d46efa080c82f3a413b6902b1ff0ae"
server_key_iv: "116eec38e73dfa052d3022f5aa6e7f7a"
server_key_version: "encryptionkey@0"
I20260807 18:28:10.101298 31626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:28:10.104771 31644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:10.106009 31626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 18:28:10.106180 31626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "c4fcd8b06ef6451b9f3c2b49e615eda9"
format_stamp: "Formatted at 2026-08-07 18:28:10 on dist-test-slave-1xrh"
server_key: "c0d46efa080c82f3a413b6902b1ff0ae"
server_key_iv: "116eec38e73dfa052d3022f5aa6e7f7a"
server_key_version: "encryptionkey@0"
I20260807 18:28:10.106364 31626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:10.132895 31626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:10.136920 31626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:10.137208 31626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:10.138182 31626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:10.139529 31626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:10.139602 31626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:10.139765 31626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:10.139817 31626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:10.172811 31626 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:36591
I20260807 18:28:10.172955 31759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:36591 every 8 connection(s)
I20260807 18:28:10.174170 31626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:10.184366 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 31626
I20260807 18:28:10.184579 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:28:10.185014 27655 external_mini_cluster.cc:1428] Setting key eafe44d02226a8d98e399cba0135da84
Aug 07 18:28:10 dist-test-slave-1xrh krb5kdc[31373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:10.195722 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:10.176691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:49199 (local address 127.27.1.254:46787)
0807 18:28:10.176873 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:10.176878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:10.177972 (+ 1094us) server_negotiation.cc:407] Connection header received
0807 18:28:10.179782 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:10.179787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:10.179857 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:10.179951 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:10.181313 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.182032 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.183546 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.183984 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.184077 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:10.188092 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:10.188102 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:10.188110 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:10.188160 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:10.192305 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.193024 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.193033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.193040 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.193222 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.193649 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.193657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.193663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.194073 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:10.194243 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:10.194906 (+ 663us) server_negotiation.cc:299] Negotiation successful
0807 18:28:10.195214 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 18:28:10.196274 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:10.177129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:46787 (local address 127.27.1.194:49199)
0807 18:28:10.177812 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:10.177849 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:10.179547 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:10.180150 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:10.180158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:10.180623 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:10.181170 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:10.181187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.182156 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.182159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:10.183423 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:10.183440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.184351 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.184362 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:10.184691 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:10.185396 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:10.185418 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:10.187846 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:10.192445 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:10.192452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:10.192464 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:10.192881 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:10.193338 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:10.193341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:10.193343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:10.193529 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:10.194196 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:10.194202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:10.194514 (+ 312us) client_negotiation.cc:769] Sending connection context
0807 18:28:10.194742 (+ 228us) client_negotiation.cc:240] Negotiation successful
0807 18:28:10.195098 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":272,"threads_started":1}
I20260807 18:28:10.197789 31760 heartbeater.cc:347] Connected to a master server at 127.27.1.254:46787
I20260807 18:28:10.198179 31760 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:10.198948 31760 heartbeater.cc:511] Master 127.27.1.254:46787 requested a full tablet report, sending...
I20260807 18:28:10.200711 31420 ts_manager.cc:194] Registered new tserver with Master: c4fcd8b06ef6451b9f3c2b49e615eda9 (127.27.1.194:36591)
I20260807 18:28:10.201661 31420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:49199
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:28:10.238678 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46787
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:10.580359 31765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:10.580669 31765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:10.580845 31765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:10.587927 31765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:10.588059 31765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:10.588220 31765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:10.597770 31765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:10.599169 31765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:10.600576 31765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:10.613857 31773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:10.613893 31772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:10.613957 31775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:10.614537 31765 server_base.cc:1034] running on GCE node
I20260807 18:28:10.615170 31765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:10.616137 31765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:10.617406 31765 hybrid_clock.cc:648] HybridClock initialized: now 1786127290617323 us; error 39 us; skew 500 ppm
I20260807 18:28:10.617714 31765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:10 dist-test-slave-1xrh krb5kdc[31373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:10.621349 31765 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:28:10.623019 31765 webserver.cc:460] Webserver started at http://127.27.1.195:36631/ using document root <none> and password file <none>
I20260807 18:28:10.623793 31765 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:10.624037 31765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:10.624385 31765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:10.627638 31765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "a9e5ba3282f143219c64fd6ef1ec0310"
format_stamp: "Formatted at 2026-08-07 18:28:10 on dist-test-slave-1xrh"
server_key: "ad8fd2608af98f7b0b9086ce1bef3405"
server_key_iv: "b17e6e55f357333bc214bd10cba3421c"
server_key_version: "encryptionkey@0"
I20260807 18:28:10.628520 31765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "a9e5ba3282f143219c64fd6ef1ec0310"
format_stamp: "Formatted at 2026-08-07 18:28:10 on dist-test-slave-1xrh"
server_key: "ad8fd2608af98f7b0b9086ce1bef3405"
server_key_iv: "b17e6e55f357333bc214bd10cba3421c"
server_key_version: "encryptionkey@0"
I20260807 18:28:10.633869 31765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 18:28:10.637660 31782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:10.638990 31765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:28:10.639221 31765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "a9e5ba3282f143219c64fd6ef1ec0310"
format_stamp: "Formatted at 2026-08-07 18:28:10 on dist-test-slave-1xrh"
server_key: "ad8fd2608af98f7b0b9086ce1bef3405"
server_key_iv: "b17e6e55f357333bc214bd10cba3421c"
server_key_version: "encryptionkey@0"
I20260807 18:28:10.639436 31765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:10.669371 31765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:10.673750 31765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:10.674049 31765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:10.675081 31765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:10.676806 31765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:10.676879 31765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:10.676963 31765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:10.677023 31765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:10.700556 31619 heartbeater.cc:503] Master 127.27.1.254:46787 was elected leader, sending a full tablet report...
I20260807 18:28:10.712700 31765 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:38567
I20260807 18:28:10.712739 31902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:38567 every 8 connection(s)
I20260807 18:28:10.714005 31765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:10.719597 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 31765
I20260807 18:28:10.719798 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:28:10.720366 27655 external_mini_cluster.cc:1428] Setting key 87a5f84aa0d3a55121baace431c51e2f
Aug 07 18:28:10 dist-test-slave-1xrh krb5kdc[31373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:10.734656 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:10.716488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:43165 (local address 127.27.1.254:46787)
0807 18:28:10.717045 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:10.717052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:10.717856 (+ 804us) server_negotiation.cc:407] Connection header received
0807 18:28:10.720137 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:10.720141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:10.720210 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:10.720305 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:10.722242 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.723455 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.725041 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.725429 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.725522 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:10.729120 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:10.729125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:10.729128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:10.729160 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:10.731864 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.732529 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.732532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.732535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.732649 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.733028 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.733031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.733033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.733279 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:10.733388 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:10.734079 (+ 691us) server_negotiation.cc:299] Negotiation successful
0807 18:28:10.734319 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":276,"threads_started":1}
I20260807 18:28:10.735790 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:10.716957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:46787 (local address 127.27.1.195:43165)
0807 18:28:10.717707 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:10.717748 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:10.719840 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:10.720586 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:10.720597 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:10.721287 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:10.722054 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:10.722082 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.723622 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.723627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:10.724918 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:10.724931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.725615 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.725620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:10.725812 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:10.726458 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:10.726483 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:10.728926 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:10.732000 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:10.732006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:10.732019 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:10.732432 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:10.732740 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:10.732743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:10.732745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:10.732935 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:10.733399 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:10.733405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:10.733695 (+ 290us) client_negotiation.cc:769] Sending connection context
0807 18:28:10.733999 (+ 304us) client_negotiation.cc:240] Negotiation successful
0807 18:28:10.734490 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":270,"threads_started":1}
I20260807 18:28:10.737159 31903 heartbeater.cc:347] Connected to a master server at 127.27.1.254:46787
I20260807 18:28:10.737434 31903 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:10.738094 31903 heartbeater.cc:511] Master 127.27.1.254:46787 requested a full tablet report, sending...
I20260807 18:28:10.739460 31420 ts_manager.cc:194] Registered new tserver with Master: a9e5ba3282f143219c64fd6ef1ec0310 (127.27.1.195:38567)
I20260807 18:28:10.740247 31420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:43165
I20260807 18:28:10.748306 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 18:28:10 dist-test-slave-1xrh krb5kdc[31373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 18:28:10 dist-test-slave-1xrh krb5kdc[31373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:28:10.809378 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:10.786834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.27.1.193:34125)
0807 18:28:10.787509 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:10.787519 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:10.787554 (+ 35us) server_negotiation.cc:407] Connection header received
0807 18:28:10.787640 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:10.787647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:10.787871 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:10.788050 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:10.789503 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.791077 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.792657 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.793512 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.793612 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:10.803542 (+ 9930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:10.803554 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:10.803570 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:10.803610 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:10.806340 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.806974 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.806982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.806984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.807107 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.807535 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.807543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.807549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.808160 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:10.808510 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:10.808727 (+ 217us) server_negotiation.cc:299] Negotiation successful
0807 18:28:10.808974 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":313,"threads_started":1}
Aug 07 18:28:10 dist-test-slave-1xrh krb5kdc[31373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:10.828284 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:10.814594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.27.1.254:46787)
0807 18:28:10.814831 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:10.814837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:10.814943 (+ 106us) server_negotiation.cc:407] Connection header received
0807 18:28:10.815080 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:10.815084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:10.815194 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:10.815267 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:10.816529 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.817479 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.818772 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.819221 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.819313 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:10.822991 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:10.822995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:10.822999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:10.823030 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:10.825693 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.826334 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.826341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.826344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.826471 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.826836 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.826841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.826845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.827213 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:10.827461 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:10.827741 (+ 280us) server_negotiation.cc:299] Negotiation successful
0807 18:28:10.827965 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260807 18:28:10.828833 31420 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45056
I20260807 18:28:10.829820 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 31485
I20260807 18:28:10.842020 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:10.830850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.27.1.193:34125)
0807 18:28:10.831061 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:10.831065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:10.831158 (+ 93us) server_negotiation.cc:407] Connection header received
0807 18:28:10.831320 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:10.831324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:10.831397 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:10.831471 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:10.832618 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.833284 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.834379 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.834856 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.834955 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:10.837186 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:10.837191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:10.837194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:10.837225 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:10.839755 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.840314 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.840318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.840320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.840436 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.840740 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.840742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.840744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.841013 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:10.841186 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:10.841496 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 18:28:10.841649 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 18:28:10.842682 31614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:10.931921 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 31485
I20260807 18:28:10.957439 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 31626
Aug 07 18:28:10 dist-test-slave-1xrh krb5kdc[31373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:28:10.977103 31918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:10.958381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.27.1.194:36591)
0807 18:28:10.959016 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:10.959029 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:10.959069 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:28:10.959179 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:10.959190 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:10.959525 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:10.959722 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:10.961351 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.963195 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.965082 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:10.966139 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:10.966294 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:10.971668 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:10.971680 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:10.971694 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:10.971734 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:10.974592 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.975203 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.975210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.975212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.975342 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:10.975668 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:10.975671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:10.975673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:10.976033 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:10.976247 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:10.976503 (+ 256us) server_negotiation.cc:299] Negotiation successful
0807 18:28:10.976693 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":279,"threads_started":1}
I20260807 18:28:10.978384 31755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:11.061800 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 31626
I20260807 18:28:11.081815 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 31765
Aug 07 18:28:11 dist-test-slave-1xrh krb5kdc[31373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:28:11.097873 31920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:11.082774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.27.1.195:38567)
0807 18:28:11.083411 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:11.083418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:11.083447 (+ 29us) server_negotiation.cc:407] Connection header received
0807 18:28:11.083516 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:11.083522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:11.083729 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:11.083913 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:11.085256 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:11.086308 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:11.087649 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:11.088233 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:11.088384 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:11.092556 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:11.092566 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:11.092581 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:11.092613 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:11.095445 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:11.096054 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:11.096062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:11.096064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:11.096178 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:11.096465 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:11.096468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:11.096470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:11.096780 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:11.097026 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:11.097299 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 18:28:11.097573 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":275,"threads_started":1}
I20260807 18:28:11.099287 31898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:11.185343 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 31765
I20260807 18:28:11.205840 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 31387
I20260807 18:28:11.207187 31452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:11.311769 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 31387
2026-08-07T18:28:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4128 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:28:11 dist-test-slave-1xrh krb5kdc[31925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:28:11 dist-test-slave-1xrh krb5kdc[31925](info): set up 2 sockets
Aug 07 18:28:11 dist-test-slave-1xrh krb5kdc[31925](info): commencing operation
krb5kdc: starting...
W20260807 18:28:12.533605 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:28:12 dist-test-slave-1xrh krb5kdc[31925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:28:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:28:12.754714 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34315
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:34315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:13.131325 31941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:13.131662 31941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:13.131817 31941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:13.138999 31941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:13.139148 31941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:13.139261 31941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:13.139355 31941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:13.139437 31941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:13.149183 31941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:34315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:13.150445 31941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:13.151844 31941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:13.163355 31946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:13.163841 31949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:13.163890 31947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:13.164391 31941 server_base.cc:1034] running on GCE node
I20260807 18:28:13.165184 31941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:13.166457 31941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:13.167692 31941 hybrid_clock.cc:648] HybridClock initialized: now 1786127293167650 us; error 49 us; skew 500 ppm
I20260807 18:28:13.168092 31941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:13 dist-test-slave-1xrh krb5kdc[31925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:13.171226 31941 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:28:13.172307 31941 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:13.172446 31941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:13.172770 31941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:13.175818 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "a53222f4b3cb466fb7b03abf3f5caa55"
format_stamp: "Formatted at 2026-08-07 18:28:13 on dist-test-slave-1xrh"
server_key: "6659bea71d32a9d09213c6e16252bbc3"
server_key_iv: "8f7f027436c0634f776f03c876fce6f8"
server_key_version: "encryptionkey@0"
I20260807 18:28:13.176636 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "a53222f4b3cb466fb7b03abf3f5caa55"
format_stamp: "Formatted at 2026-08-07 18:28:13 on dist-test-slave-1xrh"
server_key: "6659bea71d32a9d09213c6e16252bbc3"
server_key_iv: "8f7f027436c0634f776f03c876fce6f8"
server_key_version: "encryptionkey@0"
I20260807 18:28:13.181524 31941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:28:13.184901 31955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:13.186164 31941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:28:13.186349 31941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "a53222f4b3cb466fb7b03abf3f5caa55"
format_stamp: "Formatted at 2026-08-07 18:28:13 on dist-test-slave-1xrh"
server_key: "6659bea71d32a9d09213c6e16252bbc3"
server_key_iv: "8f7f027436c0634f776f03c876fce6f8"
server_key_version: "encryptionkey@0"
I20260807 18:28:13.186568 31941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:13.215217 31941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:13.219203 31941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:13.219486 31941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:13.238399 31941 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:34315
I20260807 18:28:13.238418 32009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:34315 every 8 connection(s)
I20260807 18:28:13.239303 31941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:13.242887 32010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:13.246575 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 31941
I20260807 18:28:13.246826 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:13.247542 27655 external_mini_cluster.cc:1428] Setting key 4c73948d371883fab839eccb487891e9
I20260807 18:28:13.254166 32010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55: Bootstrap starting.
I20260807 18:28:13.257855 32010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:13.259102 32010 log.cc:824] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:13.261992 32010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55: No bootstrap required, opened a new log
Aug 07 18:28:13 dist-test-slave-1xrh krb5kdc[31925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:13.270313 32010 raft_consensus.cc:348] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53222f4b3cb466fb7b03abf3f5caa55" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34315 } }
I20260807 18:28:13.270704 32010 raft_consensus.cc:374] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:13.270819 32010 raft_consensus.cc:729] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53222f4b3cb466fb7b03abf3f5caa55, State: Initialized, Role: FOLLOWER
I20260807 18:28:13.271409 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:13.249680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.27.1.254:34315)
0807 18:28:13.250718 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:13.250729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:13.250772 (+ 43us) server_negotiation.cc:407] Connection header received
0807 18:28:13.252653 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:13.252676 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:13.253047 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:13.253374 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:13.255349 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:13.256769 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:13.258663 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:13.259551 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:13.259753 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:13.264053 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:13.264063 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:13.264076 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:13.264118 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:13.267300 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:13.267950 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:13.267958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:13.267966 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:13.268144 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:13.268562 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:13.268567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:13.268571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:13.269127 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:13.269446 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:13.269864 (+ 418us) server_negotiation.cc:299] Negotiation successful
0807 18:28:13.270260 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":398,"threads_started":1}
I20260807 18:28:13.271482 32010 consensus_queue.cc:260] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53222f4b3cb466fb7b03abf3f5caa55" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34315 } }
I20260807 18:28:13.271795 32010 raft_consensus.cc:388] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:13.271958 32010 raft_consensus.cc:482] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:13.272161 32010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:13.274571 32010 raft_consensus.cc:504] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53222f4b3cb466fb7b03abf3f5caa55" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34315 } }
I20260807 18:28:13.275058 32010 leader_election.cc:302] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a53222f4b3cb466fb7b03abf3f5caa55; no voters:
I20260807 18:28:13.275739 32010 leader_election.cc:288] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:13.275851 32015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:13.277771 32010 sys_catalog.cc:564] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:13.279513 32015 raft_consensus.cc:686] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [term 1 LEADER]: Becoming Leader. State: Replica: a53222f4b3cb466fb7b03abf3f5caa55, State: Running, Role: LEADER
I20260807 18:28:13.280161 32015 consensus_queue.cc:237] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53222f4b3cb466fb7b03abf3f5caa55" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34315 } }
I20260807 18:28:13.282754 32017 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a53222f4b3cb466fb7b03abf3f5caa55. Latest consensus state: current_term: 1 leader_uuid: "a53222f4b3cb466fb7b03abf3f5caa55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53222f4b3cb466fb7b03abf3f5caa55" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34315 } } }
I20260807 18:28:13.283149 32017 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:13.283449 32016 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a53222f4b3cb466fb7b03abf3f5caa55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53222f4b3cb466fb7b03abf3f5caa55" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34315 } } }
I20260807 18:28:13.284478 32016 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55 [sys.catalog]: This master's current role is: LEADER
W20260807 18:28:13.287366 32031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 18:28:13.287482 32031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 18:28:13.287631 32032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:13.289204 32032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:13.297194 32032 catalog_manager.cc:1347] Generated new cluster ID: 2c79f2e13b4e4c12b26768e8f714c6a9
I20260807 18:28:13.297291 32032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:13.317138 32032 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:13.318467 32032 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:13.328171 32032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55: Generated new TSK 0
I20260807 18:28:13.329015 32032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:28:13.388204 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34315
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:13.720757 32038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:13.721063 32038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:13.721279 32038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:13.728562 32038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:13.728668 32038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:13.728884 32038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:13.738493 32038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:13.739899 32038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:13.741268 32038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:13.754153 32043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:13.754189 32044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:13.754087 32046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:13.755096 32038 server_base.cc:1034] running on GCE node
I20260807 18:28:13.755669 32038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:13.756542 32038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:13.757766 32038 hybrid_clock.cc:648] HybridClock initialized: now 1786127293757724 us; error 51 us; skew 500 ppm
I20260807 18:28:13.758078 32038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:13 dist-test-slave-1xrh krb5kdc[31925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:13.761368 32038 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:28:13.762522 32038 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:13.762620 32038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:13.762889 32038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:13.765838 32038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "c842aaf8a77444378a4e8df0c23afe1e"
format_stamp: "Formatted at 2026-08-07 18:28:13 on dist-test-slave-1xrh"
server_key: "aea208023ac5bb56e5eba143d6aeff56"
server_key_iv: "a81550dcea6c2627f7359237f404652f"
server_key_version: "encryptionkey@0"
I20260807 18:28:13.766607 32038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "c842aaf8a77444378a4e8df0c23afe1e"
format_stamp: "Formatted at 2026-08-07 18:28:13 on dist-test-slave-1xrh"
server_key: "aea208023ac5bb56e5eba143d6aeff56"
server_key_iv: "a81550dcea6c2627f7359237f404652f"
server_key_version: "encryptionkey@0"
I20260807 18:28:13.771656 32038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 18:28:13.775066 32052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:13.776355 32038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:28:13.776551 32038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "c842aaf8a77444378a4e8df0c23afe1e"
format_stamp: "Formatted at 2026-08-07 18:28:13 on dist-test-slave-1xrh"
server_key: "aea208023ac5bb56e5eba143d6aeff56"
server_key_iv: "a81550dcea6c2627f7359237f404652f"
server_key_version: "encryptionkey@0"
I20260807 18:28:13.776758 32038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:13.800551 32038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:13.805769 32038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:13.806057 32038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:13.807075 32038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:13.808434 32038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:13.808547 32038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:13.808670 32038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:13.808748 32038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:13.841811 32038 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:35859
I20260807 18:28:13.841897 32169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:35859 every 8 connection(s)
I20260807 18:28:13.842670 32038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:13.850833 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 32038
I20260807 18:28:13.851117 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:28:13.851955 27655 external_mini_cluster.cc:1428] Setting key 8488222810ef917ccfc18b69fc84d57c
Aug 07 18:28:13 dist-test-slave-1xrh krb5kdc[31925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:13.864557 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:13.845202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:56057 (local address 127.27.1.254:34315)
0807 18:28:13.845642 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:13.845647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:13.846604 (+ 957us) server_negotiation.cc:407] Connection header received
0807 18:28:13.848450 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:13.848454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:13.848524 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:13.848623 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:13.850070 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:13.850781 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:13.853111 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:13.853675 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:13.853811 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:13.857917 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:13.857923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:13.857926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:13.857957 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:13.861233 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:13.862182 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:13.862189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:13.862195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:13.862332 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:13.862792 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:13.862797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:13.862801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:13.863073 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:13.863283 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:13.863970 (+ 687us) server_negotiation.cc:299] Negotiation successful
0807 18:28:13.864192 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":209,"threads_started":1}
I20260807 18:28:13.865629 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:13.845724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34315 (local address 127.27.1.193:56057)
0807 18:28:13.846423 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:13.846483 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:13.848186 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:13.848815 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:13.848823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:13.849343 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:13.849910 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:13.849926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:13.850943 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:13.850950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:13.852947 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:13.852962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:13.853916 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:13.853926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:13.854170 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:13.854899 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:13.854936 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:13.857672 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:13.861380 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:13.861392 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:13.861410 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:13.862058 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:13.862478 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:13.862484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:13.862492 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:13.862664 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:13.863256 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:13.863264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:13.863587 (+ 323us) client_negotiation.cc:769] Sending connection context
0807 18:28:13.863857 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 18:28:13.864377 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":287,"threads_started":1}
I20260807 18:28:13.867136 32170 heartbeater.cc:347] Connected to a master server at 127.27.1.254:34315
I20260807 18:28:13.867466 32170 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:13.868093 32170 heartbeater.cc:511] Master 127.27.1.254:34315 requested a full tablet report, sending...
I20260807 18:28:13.870095 31973 ts_manager.cc:194] Registered new tserver with Master: c842aaf8a77444378a4e8df0c23afe1e (127.27.1.193:35859)
I20260807 18:28:13.871587 31973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:56057
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:28:13.904201 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34315
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:14.227735 32176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:14.228048 32176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:14.228214 32176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:14.235388 32176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:14.235523 32176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:14.235692 32176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:14.245376 32176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:14.246649 32176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:14.248057 32176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:14.260830 32185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:14.260723 32184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:14.260802 32187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:14.261570 32176 server_base.cc:1034] running on GCE node
I20260807 18:28:14.262135 32176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:14.263036 32176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:14.264254 32176 hybrid_clock.cc:648] HybridClock initialized: now 1786127294264200 us; error 48 us; skew 500 ppm
I20260807 18:28:14.264554 32176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:14 dist-test-slave-1xrh krb5kdc[31925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:14.267542 32176 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:28:14.268639 32176 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:14.268770 32176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:14.269081 32176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:14.271893 32176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "6f5e6e4112084e40b632da8a58d1a2a3"
format_stamp: "Formatted at 2026-08-07 18:28:14 on dist-test-slave-1xrh"
server_key: "35d1c060ef8fffd0afa9c99876268edc"
server_key_iv: "dbc3fea67fd1fbd821c03b2d7cab4999"
server_key_version: "encryptionkey@0"
I20260807 18:28:14.272810 32176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "6f5e6e4112084e40b632da8a58d1a2a3"
format_stamp: "Formatted at 2026-08-07 18:28:14 on dist-test-slave-1xrh"
server_key: "35d1c060ef8fffd0afa9c99876268edc"
server_key_iv: "dbc3fea67fd1fbd821c03b2d7cab4999"
server_key_version: "encryptionkey@0"
I20260807 18:28:14.277563 32176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 18:28:14.281201 32193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:14.282487 32176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 18:28:14.282661 32176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "6f5e6e4112084e40b632da8a58d1a2a3"
format_stamp: "Formatted at 2026-08-07 18:28:14 on dist-test-slave-1xrh"
server_key: "35d1c060ef8fffd0afa9c99876268edc"
server_key_iv: "dbc3fea67fd1fbd821c03b2d7cab4999"
server_key_version: "encryptionkey@0"
I20260807 18:28:14.282891 32176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:14.316424 32176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:14.320678 32176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:14.320966 32176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:14.322000 32176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:14.323415 32176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:14.323532 32176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:14.323664 32176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:14.323750 32176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:14.356523 32176 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:43983
I20260807 18:28:14.356606 32318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:43983 every 8 connection(s)
I20260807 18:28:14.357405 32176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:14.362643 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 32176
I20260807 18:28:14.362855 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:28:14.363446 27655 external_mini_cluster.cc:1428] Setting key 1ffbea4ac5a5d5fa8583e3b25c0ca4f6
Aug 07 18:28:14 dist-test-slave-1xrh krb5kdc[31925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:14.379720 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.359900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:41719 (local address 127.27.1.254:34315)
0807 18:28:14.360092 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:14.360096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:14.361319 (+ 1223us) server_negotiation.cc:407] Connection header received
0807 18:28:14.363502 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:14.363507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:14.363616 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:14.363735 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:14.365316 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.366062 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.368646 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.369066 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.369162 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.373260 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:14.373265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:14.373268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:14.373299 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:14.375962 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:14.376683 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:14.376687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:14.376689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:14.376806 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:14.377358 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:14.377363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:14.377372 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:14.377696 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:14.377858 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:14.379040 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0807 18:28:14.379339 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 18:28:14.380964 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.360434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34315 (local address 127.27.1.194:41719)
0807 18:28:14.361150 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:14.361196 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:14.363199 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:14.364027 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:14.364040 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:14.364605 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:14.365177 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:14.365194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.366230 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.366238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:14.368493 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:14.368510 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.369289 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.369299 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:14.369612 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.370353 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:14.370386 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:14.373048 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:14.376101 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:14.376110 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:14.376126 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:14.376554 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:14.377013 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:14.377020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:14.377022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:14.377207 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:14.377830 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:14.377837 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:14.378426 (+ 589us) client_negotiation.cc:769] Sending connection context
0807 18:28:14.378924 (+ 498us) client_negotiation.cc:240] Negotiation successful
0807 18:28:14.379508 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":288,"threads_started":1}
I20260807 18:28:14.382637 32319 heartbeater.cc:347] Connected to a master server at 127.27.1.254:34315
I20260807 18:28:14.382988 32319 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:14.383930 32319 heartbeater.cc:511] Master 127.27.1.254:34315 requested a full tablet report, sending...
I20260807 18:28:14.385579 31973 ts_manager.cc:194] Registered new tserver with Master: 6f5e6e4112084e40b632da8a58d1a2a3 (127.27.1.194:43983)
I20260807 18:28:14.386478 31973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:41719
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:28:14.415933 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34315
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:14.746280 32325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:14.746547 32325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:14.746706 32325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:14.754150 32325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:14.754251 32325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:14.754438 32325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:14.764137 32325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:14.765450 32325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:14.766855 32325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:14.779997 32333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:14.780042 32331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:14.780215 32330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:14.781586 32325 server_base.cc:1034] running on GCE node
I20260807 18:28:14.782186 32325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:14.783155 32325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:14.784404 32325 hybrid_clock.cc:648] HybridClock initialized: now 1786127294784336 us; error 61 us; skew 500 ppm
I20260807 18:28:14.784722 32325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:14 dist-test-slave-1xrh krb5kdc[31925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:14.788143 32325 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:28:14.789314 32325 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:14.789407 32325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:14.789727 32325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:14.792840 32325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "5c01d32973d344359bd683b05eb023a0"
format_stamp: "Formatted at 2026-08-07 18:28:14 on dist-test-slave-1xrh"
server_key: "1e6e135cd21b8cbff177a7af3a0779f8"
server_key_iv: "6c740f32fccbaa574e35bbcfd9ae217e"
server_key_version: "encryptionkey@0"
I20260807 18:28:14.793694 32325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "5c01d32973d344359bd683b05eb023a0"
format_stamp: "Formatted at 2026-08-07 18:28:14 on dist-test-slave-1xrh"
server_key: "1e6e135cd21b8cbff177a7af3a0779f8"
server_key_iv: "6c740f32fccbaa574e35bbcfd9ae217e"
server_key_version: "encryptionkey@0"
I20260807 18:28:14.798772 32325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 18:28:14.802232 32340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:14.803594 32325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260807 18:28:14.803799 32325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "5c01d32973d344359bd683b05eb023a0"
format_stamp: "Formatted at 2026-08-07 18:28:14 on dist-test-slave-1xrh"
server_key: "1e6e135cd21b8cbff177a7af3a0779f8"
server_key_iv: "6c740f32fccbaa574e35bbcfd9ae217e"
server_key_version: "encryptionkey@0"
I20260807 18:28:14.803998 32325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:14.834406 32325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:14.838388 32325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:14.838663 32325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:14.839653 32325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:14.840912 32325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:14.841019 32325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:14.841188 32325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:14.841276 32325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:14.874364 32170 heartbeater.cc:503] Master 127.27.1.254:34315 was elected leader, sending a full tablet report...
I20260807 18:28:14.874466 32325 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:34691
I20260807 18:28:14.874483 32464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:34691 every 8 connection(s)
I20260807 18:28:14.875379 32325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:14.877796 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 32325
I20260807 18:28:14.878021 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:28:14.878474 27655 external_mini_cluster.cc:1428] Setting key 34443976f831a695db5d8d85102d53d2
Aug 07 18:28:14 dist-test-slave-1xrh krb5kdc[31925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:14.895301 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.877931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:39105 (local address 127.27.1.254:34315)
0807 18:28:14.878163 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:14.878171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:14.879591 (+ 1420us) server_negotiation.cc:407] Connection header received
0807 18:28:14.881702 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:14.881708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:14.881837 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:14.881955 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:14.883520 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.884280 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.885786 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.886237 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.886362 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.889707 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:14.889718 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:14.889721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:14.889751 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:14.892485 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:14.893330 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:14.893335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:14.893339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:14.893491 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:14.893909 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:14.893912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:14.893914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:14.894180 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:14.894289 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:14.894860 (+ 571us) server_negotiation.cc:299] Negotiation successful
0807 18:28:14.895007 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 18:28:14.896188 32467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.878463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34315 (local address 127.27.1.195:39105)
0807 18:28:14.879439 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:14.879486 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:14.881397 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:14.882180 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:14.882188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:14.882719 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:14.883351 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:14.883370 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.884436 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.884441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:14.885647 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:14.885658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.886452 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.886461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:14.886684 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.887398 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:14.887425 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:14.889516 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:14.892621 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:14.892629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:14.892644 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:14.893209 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:14.893628 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:14.893631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:14.893633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:14.893805 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:14.894281 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:14.894287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:14.894556 (+ 269us) client_negotiation.cc:769] Sending connection context
0807 18:28:14.894771 (+ 215us) client_negotiation.cc:240] Negotiation successful
0807 18:28:14.895181 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":460,"threads_started":1}
I20260807 18:28:14.897437 32465 heartbeater.cc:347] Connected to a master server at 127.27.1.254:34315
I20260807 18:28:14.897730 32465 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:14.898331 32465 heartbeater.cc:511] Master 127.27.1.254:34315 requested a full tablet report, sending...
I20260807 18:28:14.899660 31973 ts_manager.cc:194] Registered new tserver with Master: 5c01d32973d344359bd683b05eb023a0 (127.27.1.195:34691)
I20260807 18:28:14.900439 31973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:39105
I20260807 18:28:14.905555 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:28:14.922551 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.910081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.27.1.254:34315)
0807 18:28:14.910248 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:14.910252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:14.910871 (+ 619us) server_negotiation.cc:407] Connection header received
0807 18:28:14.911064 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:14.911069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:14.911174 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:14.911278 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:14.912692 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.913405 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.914698 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.915196 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.915325 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.917551 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:14.917557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:14.917567 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:14.917598 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:14.919943 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:14.920598 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:14.920605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:14.920609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:14.920749 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:14.921112 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:14.921119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:14.921123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:14.921434 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:14.921701 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:14.922024 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 18:28:14.922239 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 18:28:14.926584 31973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:28:14.933344 31973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:28:14.958659 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.945072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35859 (local address 127.0.0.1:59804)
0807 18:28:14.945956 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:14.946006 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:14.946183 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:14.947677 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:14.947689 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:14.947741 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:14.948224 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:14.948241 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.951632 (+ 3391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.951642 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:14.954896 (+ 3254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:14.954919 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.957505 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.957514 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:14.957719 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.957786 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 18:28:14.957916 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 18:28:14.958143 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":481,"threads_started":1}
I20260807 18:28:14.959897 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.945155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.27.1.193:35859)
0807 18:28:14.946827 (+ 1672us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:14.946835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:14.946866 (+ 31us) server_negotiation.cc:407] Connection header received
0807 18:28:14.946988 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:14.947000 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:14.947355 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:14.948478 (+ 1123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:14.948804 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.951418 (+ 2614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.955217 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.957248 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.958934 (+ 1686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.959230 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:14.959308 (+ 78us) server_negotiation.cc:299] Negotiation successful
0807 18:28:14.959510 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1498,"thread_start_us":1357,"threads_started":1}
I20260807 18:28:14.963958 32100 tablet_service.cc:1467] Processing CreateTablet for tablet 4df84f180a754f0dade69a6c873d87de (DEFAULT_TABLE table=test-table [id=44e434606b4c4e18a1162936c3b0cf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:14.965407 32100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df84f180a754f0dade69a6c873d87de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:14.965802 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.945139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:43983 (local address 127.0.0.1:33604)
0807 18:28:14.956575 (+ 11436us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:14.956624 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:14.956766 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:14.958549 (+ 1783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:14.958553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:14.958576 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:14.959284 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:14.959300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.961261 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.961266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:14.963404 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:14.963416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.965035 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.965043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:14.965205 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.965246 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 18:28:14.965335 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 18:28:14.965486 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11275,"thread_start_us":13154,"threads_started":1}
I20260807 18:28:14.965952 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.945979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.27.1.194:43983)
0807 18:28:14.957353 (+ 11374us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:14.957366 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:14.957411 (+ 45us) server_negotiation.cc:407] Connection header received
0807 18:28:14.957526 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:14.957535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:14.957885 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:14.958112 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:14.959503 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.961098 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.963552 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.964791 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.964944 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.965132 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:14.965443 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 18:28:14.965615 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11174,"thread_start_us":11039,"threads_started":1}
I20260807 18:28:14.966727 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.947480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:34691 (local address 127.0.0.1:35236)
0807 18:28:14.958236 (+ 10756us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:14.958256 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:14.958382 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:14.959374 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:14.959379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:14.959403 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:14.959775 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:14.959786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.961718 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.961725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:14.963936 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:14.963954 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.965970 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.965979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:14.966139 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.966179 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 18:28:14.966267 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 18:28:14.966413 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10615,"thread_start_us":10460,"threads_started":1}
I20260807 18:28:14.967525 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:14.948338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.27.1.195:34691)
0807 18:28:14.957780 (+ 9442us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:14.957788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:14.958525 (+ 737us) server_negotiation.cc:407] Connection header received
0807 18:28:14.958642 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:14.958651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:14.958851 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:14.959060 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:14.959986 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.961566 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.964111 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:14.965487 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:14.965604 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:14.965815 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:14.966998 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0807 18:28:14.967188 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9264,"thread_start_us":9082,"threads_started":1}
I20260807 18:28:14.969619 32245 tablet_service.cc:1467] Processing CreateTablet for tablet 4df84f180a754f0dade69a6c873d87de (DEFAULT_TABLE table=test-table [id=44e434606b4c4e18a1162936c3b0cf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:14.970702 32245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df84f180a754f0dade69a6c873d87de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:14.970886 32393 tablet_service.cc:1467] Processing CreateTablet for tablet 4df84f180a754f0dade69a6c873d87de (DEFAULT_TABLE table=test-table [id=44e434606b4c4e18a1162936c3b0cf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:14.972205 32393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df84f180a754f0dade69a6c873d87de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:14.981612 32484 tablet_bootstrap.cc:492] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e: Bootstrap starting.
I20260807 18:28:14.983894 32485 tablet_bootstrap.cc:492] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: Bootstrap starting.
I20260807 18:28:14.985824 32484 tablet_bootstrap.cc:654] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:14.986505 32486 tablet_bootstrap.cc:492] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: Bootstrap starting.
I20260807 18:28:14.987597 32484 log.cc:824] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:14.988049 32485 tablet_bootstrap.cc:654] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:14.988956 32485 log.cc:824] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:14.990422 32486 tablet_bootstrap.cc:654] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:14.991868 32485 tablet_bootstrap.cc:492] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: No bootstrap required, opened a new log
I20260807 18:28:14.991928 32486 log.cc:824] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:14.992177 32485 ts_tablet_manager.cc:1397] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260807 18:28:14.997252 32484 tablet_bootstrap.cc:492] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e: No bootstrap required, opened a new log
I20260807 18:28:14.997543 32484 ts_tablet_manager.cc:1397] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260807 18:28:14.997877 32486 tablet_bootstrap.cc:492] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: No bootstrap required, opened a new log
I20260807 18:28:14.998139 32486 ts_tablet_manager.cc:1397] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260807 18:28:14.999996 32485 raft_consensus.cc:348] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 } } peers { permanent_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43983 } } peers { permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 } }
I20260807 18:28:15.000363 32485 raft_consensus.cc:374] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:15.000491 32485 raft_consensus.cc:729] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f5e6e4112084e40b632da8a58d1a2a3, State: Initialized, Role: FOLLOWER
I20260807 18:28:15.001071 32485 consensus_queue.cc:260] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 } } peers { permanent_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43983 } } peers { permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 } }
I20260807 18:28:15.002163 32319 heartbeater.cc:503] Master 127.27.1.254:34315 was elected leader, sending a full tablet report...
I20260807 18:28:15.002827 32485 ts_tablet_manager.cc:1428] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260807 18:28:15.008298 32484 raft_consensus.cc:348] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 } } peers { permanent_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43983 } } peers { permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 } }
I20260807 18:28:15.008766 32484 raft_consensus.cc:374] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:15.008896 32484 raft_consensus.cc:729] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c842aaf8a77444378a4e8df0c23afe1e, State: Initialized, Role: FOLLOWER
I20260807 18:28:15.008785 32486 raft_consensus.cc:348] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 } } peers { permanent_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43983 } } peers { permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 } }
I20260807 18:28:15.009192 32486 raft_consensus.cc:374] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:15.009295 32486 raft_consensus.cc:729] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c01d32973d344359bd683b05eb023a0, State: Initialized, Role: FOLLOWER
I20260807 18:28:15.009557 32484 consensus_queue.cc:260] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 } } peers { permanent_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43983 } } peers { permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 } }
I20260807 18:28:15.009861 32486 consensus_queue.cc:260] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 } } peers { permanent_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43983 } } peers { permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 } }
I20260807 18:28:15.011394 32465 heartbeater.cc:503] Master 127.27.1.254:34315 was elected leader, sending a full tablet report...
I20260807 18:28:15.011473 32484 ts_tablet_manager.cc:1428] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260807 18:28:15.011583 32486 ts_tablet_manager.cc:1428] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
W20260807 18:28:15.095009 32171 tablet.cc:2369] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:28:15.109215 32320 tablet.cc:2369] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:28:15.126989 32466 tablet.cc:2369] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:28:15.271900 32490 raft_consensus.cc:482] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:15.272240 32490 raft_consensus.cc:504] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 } } peers { permanent_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43983 } } peers { permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 } }
I20260807 18:28:15.273978 32490 leader_election.cc:288] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c842aaf8a77444378a4e8df0c23afe1e (127.27.1.193:35859), 5c01d32973d344359bd683b05eb023a0 (127.27.1.195:34691)
I20260807 18:28:15.283223 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.274480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:36963 (local address 127.27.1.195:34691)
0807 18:28:15.274705 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:15.274713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:15.275330 (+ 617us) server_negotiation.cc:407] Connection header received
0807 18:28:15.275559 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:15.275565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:15.275645 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:15.275747 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:15.276623 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.277799 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.280758 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.282062 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.282230 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:15.282302 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:15.282810 (+ 508us) server_negotiation.cc:299] Negotiation successful
0807 18:28:15.282939 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 18:28:15.283223 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.274523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:43111 (local address 127.27.1.193:35859)
0807 18:28:15.274769 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:15.274779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:15.275376 (+ 597us) server_negotiation.cc:407] Connection header received
0807 18:28:15.275565 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:15.275570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:15.275645 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:15.275747 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:15.276679 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.277748 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.280734 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.282062 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.282230 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:15.282302 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:15.282810 (+ 508us) server_negotiation.cc:299] Negotiation successful
0807 18:28:15.282939 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 18:28:15.283468 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.274252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:34691 (local address 127.27.1.194:36963)
0807 18:28:15.275202 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:15.275227 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:15.275391 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:15.275779 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:15.275784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:15.275835 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:15.276449 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:15.276473 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.277935 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.277942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:15.280580 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:15.280595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.282343 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.282350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:15.282603 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:15.282670 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 18:28:15.282765 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 18:28:15.282928 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":252,"spinlock_wait_cycles":26496,"thread_start_us":645,"threads_started":1}
I20260807 18:28:15.283502 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.274282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35859 (local address 127.27.1.194:43111)
0807 18:28:15.275229 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:15.275251 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:15.275431 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:15.275779 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:15.275784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:15.275812 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:15.276525 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:15.276538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.277883 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.277888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:15.280581 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:15.280595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.282347 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.282351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:15.282597 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:15.282667 (+ 70us) client_negotiation.cc:769] Sending connection context
0807 18:28:15.282765 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 18:28:15.282928 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":222,"spinlock_wait_cycles":43392,"thread_start_us":520,"threads_started":1}
I20260807 18:28:15.284626 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df84f180a754f0dade69a6c873d87de" candidate_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c842aaf8a77444378a4e8df0c23afe1e" is_pre_election: true
I20260807 18:28:15.284725 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df84f180a754f0dade69a6c873d87de" candidate_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c01d32973d344359bd683b05eb023a0" is_pre_election: true
I20260807 18:28:15.285156 32122 raft_consensus.cc:2454] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f5e6e4112084e40b632da8a58d1a2a3 in term 0.
I20260807 18:28:15.285199 32415 raft_consensus.cc:2454] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f5e6e4112084e40b632da8a58d1a2a3 in term 0.
I20260807 18:28:15.285848 32199 leader_election.cc:302] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5c01d32973d344359bd683b05eb023a0, 6f5e6e4112084e40b632da8a58d1a2a3, c842aaf8a77444378a4e8df0c23afe1e; no voters:
I20260807 18:28:15.286290 32490 raft_consensus.cc:2781] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:28:15.286443 32490 raft_consensus.cc:482] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:28:15.286559 32490 raft_consensus.cc:3037] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:15.288872 32490 raft_consensus.cc:504] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 } } peers { permanent_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43983 } } peers { permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 } }
I20260807 18:28:15.289529 32490 leader_election.cc:288] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [CANDIDATE]: Term 1 election: Requested vote from peers c842aaf8a77444378a4e8df0c23afe1e (127.27.1.193:35859), 5c01d32973d344359bd683b05eb023a0 (127.27.1.195:34691)
I20260807 18:28:15.289947 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df84f180a754f0dade69a6c873d87de" candidate_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c842aaf8a77444378a4e8df0c23afe1e"
I20260807 18:28:15.289984 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df84f180a754f0dade69a6c873d87de" candidate_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c01d32973d344359bd683b05eb023a0"
I20260807 18:28:15.290145 32122 raft_consensus.cc:3037] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:15.290160 32415 raft_consensus.cc:3037] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:15.292400 32122 raft_consensus.cc:2454] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f5e6e4112084e40b632da8a58d1a2a3 in term 1.
I20260807 18:28:15.292441 32415 raft_consensus.cc:2454] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f5e6e4112084e40b632da8a58d1a2a3 in term 1.
I20260807 18:28:15.292804 32199 leader_election.cc:302] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5c01d32973d344359bd683b05eb023a0, 6f5e6e4112084e40b632da8a58d1a2a3, c842aaf8a77444378a4e8df0c23afe1e; no voters:
I20260807 18:28:15.293087 32490 raft_consensus.cc:2781] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:15.293509 32490 raft_consensus.cc:686] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 1 LEADER]: Becoming Leader. State: Replica: 6f5e6e4112084e40b632da8a58d1a2a3, State: Running, Role: LEADER
I20260807 18:28:15.293991 32490 consensus_queue.cc:237] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 } } peers { permanent_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43983 } } peers { permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 } }
I20260807 18:28:15.298915 31972 catalog_manager.cc:5495] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f5e6e4112084e40b632da8a58d1a2a3 (127.27.1.194). New cstate: current_term: 1 leader_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f5e6e4112084e40b632da8a58d1a2a3" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 43983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 } health_report { overall_health: UNKNOWN } } }
I20260807 18:28:15.330807 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.324738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.27.1.194:43983)
0807 18:28:15.324969 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:15.324977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:15.325037 (+ 60us) server_negotiation.cc:407] Connection header received
0807 18:28:15.325242 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:15.325246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:15.325339 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:15.325465 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:15.325915 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.326956 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.328550 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.329057 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.329166 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:15.329656 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:15.330163 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:15.330276 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:15.330427 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 18:28:15.330494 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 18:28:15.338954 32415 raft_consensus.cc:1261] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0 [term 1 FOLLOWER]: Refusing update from remote peer 6f5e6e4112084e40b632da8a58d1a2a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:15.339079 32122 raft_consensus.cc:1261] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e [term 1 FOLLOWER]: Refusing update from remote peer 6f5e6e4112084e40b632da8a58d1a2a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:15.340008 32495 consensus_queue.cc:1035] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c01d32973d344359bd683b05eb023a0" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:15.340428 32490 consensus_queue.cc:1035] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c842aaf8a77444378a4e8df0c23afe1e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:15.351872 32498 mvcc.cc:205] Tried to move back new op lower bound from 7315977401698824192 to 7315977401527865344. Current Snapshot: MvccSnapshot[applied={T|T < 7315977401698824192}]
I20260807 18:28:15.370162 31972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:15.370453 31972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:15.373919 31972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a53222f4b3cb466fb7b03abf3f5caa55: Sending DeleteTablet for 3 replicas of tablet 4df84f180a754f0dade69a6c873d87de
I20260807 18:28:15.375275 32245 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df84f180a754f0dade69a6c873d87de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:15 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:33604
I20260807 18:28:15.375437 32100 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df84f180a754f0dade69a6c873d87de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:15 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:59804
I20260807 18:28:15.376838 32393 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df84f180a754f0dade69a6c873d87de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:15 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:35236
I20260807 18:28:15.381891 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 32038
I20260807 18:28:15.385676 32511 tablet_replica.cc:330] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: stopping tablet replica
I20260807 18:28:15.386188 32511 raft_consensus.cc:2229] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:28:15.386977 32511 raft_consensus.cc:2258] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:15.389637 32511 ts_tablet_manager.cc:1905] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:15.397823 32511 ts_tablet_manager.cc:1918] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:15.398061 32511 log.cc:1196] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/4df84f180a754f0dade69a6c873d87de
I20260807 18:28:15.398751 32511 ts_tablet_manager.cc:1939] T 4df84f180a754f0dade69a6c873d87de P 6f5e6e4112084e40b632da8a58d1a2a3: Deleting consensus metadata
Aug 07 18:28:15 dist-test-slave-1xrh krb5kdc[31925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:28:15.400720 31957 catalog_manager.cc:4841] TS 6f5e6e4112084e40b632da8a58d1a2a3 (127.27.1.194:43983): tablet 4df84f180a754f0dade69a6c873d87de (table test-table [id=44e434606b4c4e18a1162936c3b0cf09]) successfully deleted
I20260807 18:28:15.401655 32513 tablet_replica.cc:330] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: stopping tablet replica
I20260807 18:28:15.402179 32513 raft_consensus.cc:2229] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:15.403012 32513 raft_consensus.cc:2258] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:15.404179 32512 tablet_replica.cc:330] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e: stopping tablet replica
I20260807 18:28:15.404707 32512 raft_consensus.cc:2229] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:15.405630 32512 raft_consensus.cc:2258] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:15.405893 32513 ts_tablet_manager.cc:1905] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:15.408087 32512 ts_tablet_manager.cc:1905] T 4df84f180a754f0dade69a6c873d87de P c842aaf8a77444378a4e8df0c23afe1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:15.411827 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.383815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.27.1.193:35859)
0807 18:28:15.384055 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:15.384065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:15.388049 (+ 3984us) server_negotiation.cc:407] Connection header received
0807 18:28:15.388252 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:15.388260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:15.388383 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:15.388492 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:15.390713 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.392202 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.394561 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.395480 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.395599 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:15.402418 (+ 6819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:15.402440 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:15.402465 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:15.402525 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:15.407358 (+ 4833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:15.408226 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:15.408235 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:15.408238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:15.408473 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:15.408973 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:15.408978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:15.408982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:15.409513 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:15.409905 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:15.411178 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0807 18:28:15.411430 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 18:28:15.412772 32165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:15.413120 32513 ts_tablet_manager.cc:1918] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:15.413409 32513 log.cc:1196] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/4df84f180a754f0dade69a6c873d87de
I20260807 18:28:15.414135 32513 ts_tablet_manager.cc:1939] T 4df84f180a754f0dade69a6c873d87de P 5c01d32973d344359bd683b05eb023a0: Deleting consensus metadata
I20260807 18:28:15.417541 31958 catalog_manager.cc:4841] TS 5c01d32973d344359bd683b05eb023a0 (127.27.1.195:34691): tablet 4df84f180a754f0dade69a6c873d87de (table test-table [id=44e434606b4c4e18a1162936c3b0cf09]) successfully deleted
I20260807 18:28:15.541208 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 32038
I20260807 18:28:15.562033 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 32176
W20260807 18:28:15.562055 31959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.1.193:35859 (error 108)
I20260807 18:28:15.564706 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.563823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35859 (local address 127.0.0.1:59808)
0807 18:28:15.564321 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:15.564466 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
W20260807 18:28:15.565004 31959 catalog_manager.cc:4584] TS c842aaf8a77444378a4e8df0c23afe1e (127.27.1.193:35859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4df84f180a754f0dade69a6c873d87de: Network error: Client connection negotiation failed: client connection to 127.27.1.193:35859: connect: Connection refused (error 111)
Aug 07 18:28:15 dist-test-slave-1xrh krb5kdc[31925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:28:15.578521 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.563328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.27.1.194:43983)
0807 18:28:15.563506 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:15.563510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:15.563548 (+ 38us) server_negotiation.cc:407] Connection header received
0807 18:28:15.563605 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:15.563609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:15.563670 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:15.563789 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:15.565458 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.566282 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.567658 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.568101 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.568230 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:15.572591 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:15.572604 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:15.572619 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:15.572657 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:15.575715 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:15.576396 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:15.576403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:15.576406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:15.576549 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:15.576917 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:15.576920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:15.576922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:15.577310 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:15.577651 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:15.577890 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 18:28:15.578148 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 18:28:15.579452 32314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:15.618835 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.618364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35859 (local address 127.0.0.1:59810)
0807 18:28:15.618581 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:15.618691 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 18:28:15.619984 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.619568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35859 (local address 127.0.0.1:59820)
0807 18:28:15.619760 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:15.619828 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 18:28:15.680507 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.679905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35859 (local address 127.0.0.1:59836)
0807 18:28:15.680261 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:15.680354 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260807 18:28:15.681608 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.681248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35859 (local address 127.0.0.1:59852)
0807 18:28:15.681402 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:15.681466 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 18:28:15.695289 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 32176
I20260807 18:28:15.718256 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 32325
Aug 07 18:28:15 dist-test-slave-1xrh krb5kdc[31925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:28:15.733014 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.719030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.27.1.195:34691)
0807 18:28:15.719239 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:15.719248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:15.719392 (+ 144us) server_negotiation.cc:407] Connection header received
0807 18:28:15.719538 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:15.719542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:15.719606 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:15.719690 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:15.720957 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.721677 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.722851 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:15.723383 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:15.723472 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:15.727583 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:15.727597 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:15.727613 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:15.727650 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:15.730393 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:15.730931 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:15.730946 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:15.730948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:15.731076 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:15.731591 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:15.731594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:15.731596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:15.731946 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:15.732159 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:15.732509 (+ 350us) server_negotiation.cc:299] Negotiation successful
0807 18:28:15.732692 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 18:28:15.733668 32459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:15.762372 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.761791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35859 (local address 127.0.0.1:59856)
0807 18:28:15.762124 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:15.762218 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260807 18:28:15.763453 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:15.763022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35859 (local address 127.0.0.1:59864)
0807 18:28:15.763254 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:15.763324 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 18:28:15.824173 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 32325
I20260807 18:28:15.845261 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 31941
I20260807 18:28:15.846504 32004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:15.959070 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 31941
2026-08-07T18:28:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4646 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T18:28:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:16Z Disabled control of system clock
I20260807 18:28:16.035178 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:46181
--webserver_interface=127.27.1.254
--webserver_port=0
--builtin_ntp_servers=127.27.1.212:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:46181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 18:28:16.363039 32525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:16.363400 32525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:16.363574 32525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:16.370783 32525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:16.370887 32525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:16.370973 32525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:16.371083 32525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:16.371186 32525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:16.380424 32525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:46181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:46181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:16.381743 32525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:16.383661 32525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:16.395449 32530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:16.395673 32533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:16.395684 32531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:16.396615 32525 server_base.cc:1034] running on GCE node
I20260807 18:28:16.397357 32525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:16.398662 32525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:16.399909 32525 hybrid_clock.cc:648] HybridClock initialized: now 1786127296399845 us; error 54 us; skew 500 ppm
I20260807 18:28:16.400288 32525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:28:16.408907 32525 webserver.cc:460] Webserver started at http://127.27.1.254:35383/ using document root <none> and password file <none>
I20260807 18:28:16.409591 32525 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:16.409719 32525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:16.410049 32525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:16.413067 32525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "7499b020d72942ba9f47298c01880821"
format_stamp: "Formatted at 2026-08-07 18:28:16 on dist-test-slave-1xrh"
server_key: "6c9992013b3997360946cb89b36f655d"
server_key_iv: "985d9623396bcfce4b164215fbbc081c"
server_key_version: "encryptionkey@0"
I20260807 18:28:16.413883 32525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "7499b020d72942ba9f47298c01880821"
format_stamp: "Formatted at 2026-08-07 18:28:16 on dist-test-slave-1xrh"
server_key: "6c9992013b3997360946cb89b36f655d"
server_key_iv: "985d9623396bcfce4b164215fbbc081c"
server_key_version: "encryptionkey@0"
I20260807 18:28:16.418948 32525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 18:28:16.422175 32539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:16.423578 32525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:16.423749 32525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "7499b020d72942ba9f47298c01880821"
format_stamp: "Formatted at 2026-08-07 18:28:16 on dist-test-slave-1xrh"
server_key: "6c9992013b3997360946cb89b36f655d"
server_key_iv: "985d9623396bcfce4b164215fbbc081c"
server_key_version: "encryptionkey@0"
I20260807 18:28:16.423975 32525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:16.429991 32525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:16.430732 32525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:16.431037 32525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:16.449182 32525 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:46181
I20260807 18:28:16.449218 32595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:46181 every 8 connection(s)
I20260807 18:28:16.450512 32525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:16.453765 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 32525
I20260807 18:28:16.453974 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:16.454300 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:16.454712 27655 external_mini_cluster.cc:1428] Setting key 46b3b82b1113bd1c236ce1a399454f77
I20260807 18:28:16.463794 32599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:16.456757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.27.1.254:46181)
0807 18:28:16.457924 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:16.457935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:16.457974 (+ 39us) server_negotiation.cc:407] Connection header received
0807 18:28:16.460259 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:16.460281 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:16.460716 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:16.461047 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:16.462168 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:16.462174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:16.462185 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:16.462200 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:16.462262 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:16.462278 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:16.462464 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:16.462582 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 18:28:16.462821 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":481,"threads_started":1}
I20260807 18:28:16.466939 32596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821: Bootstrap starting.
I20260807 18:28:16.469861 32596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:16.471017 32596 log.cc:824] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:16.473626 32596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821: No bootstrap required, opened a new log
I20260807 18:28:16.480885 32596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499b020d72942ba9f47298c01880821" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46181 } }
I20260807 18:28:16.481187 32596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:16.481290 32596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7499b020d72942ba9f47298c01880821, State: Initialized, Role: FOLLOWER
I20260807 18:28:16.481894 32596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499b020d72942ba9f47298c01880821" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46181 } }
I20260807 18:28:16.482157 32596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:16.482295 32596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:16.482420 32596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:16.484714 32596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499b020d72942ba9f47298c01880821" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46181 } }
I20260807 18:28:16.485128 32596 leader_election.cc:302] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7499b020d72942ba9f47298c01880821; no voters:
I20260807 18:28:16.485672 32596 leader_election.cc:288] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:16.485808 32601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:16.486591 32601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [term 1 LEADER]: Becoming Leader. State: Replica: 7499b020d72942ba9f47298c01880821, State: Running, Role: LEADER
I20260807 18:28:16.487112 32601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499b020d72942ba9f47298c01880821" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46181 } }
I20260807 18:28:16.487712 32596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:16.490092 32602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7499b020d72942ba9f47298c01880821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499b020d72942ba9f47298c01880821" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46181 } } }
I20260807 18:28:16.490796 32602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:16.490759 32603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7499b020d72942ba9f47298c01880821. Latest consensus state: current_term: 1 leader_uuid: "7499b020d72942ba9f47298c01880821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499b020d72942ba9f47298c01880821" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46181 } } }
I20260807 18:28:16.491216 32603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:16.492784 32607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:16.497267 32607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:16.505740 32607 catalog_manager.cc:1347] Generated new cluster ID: c10ca6c65e4d4859af88b72feeb4eff1
I20260807 18:28:16.505846 32607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:16.525080 32607 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:16.525477 32607 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:16.541872 32607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821: Generated new TSK 0
I20260807 18:28:16.542757 32607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:28:16.550632 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46181
--builtin_ntp_servers=127.27.1.212:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 18:28:16.881567 32621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:16.881836 32621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:16.881995 32621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:16.889364 32621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:16.889490 32621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:16.889659 32621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:16.898824 32621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:16.900242 32621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:16.902083 32621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:16.915349 32629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:16.915349 32631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:16.915393 32628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:16.916261 32621 server_base.cc:1034] running on GCE node
I20260807 18:28:16.916842 32621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:16.917699 32621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:16.918926 32621 hybrid_clock.cc:648] HybridClock initialized: now 1786127296918880 us; error 48 us; skew 500 ppm
I20260807 18:28:16.919301 32621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:28:16.921572 32621 webserver.cc:460] Webserver started at http://127.27.1.193:33823/ using document root <none> and password file <none>
I20260807 18:28:16.922271 32621 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:16.922358 32621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:16.922736 32621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:16.925555 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "af77369e02b14bbdb040d139dbc9e20e"
format_stamp: "Formatted at 2026-08-07 18:28:16 on dist-test-slave-1xrh"
server_key: "c66c3203ea77748e08a8db6eab845217"
server_key_iv: "ce8da4be6e02410654dcf8942acc3e3a"
server_key_version: "encryptionkey@0"
I20260807 18:28:16.926283 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "af77369e02b14bbdb040d139dbc9e20e"
format_stamp: "Formatted at 2026-08-07 18:28:16 on dist-test-slave-1xrh"
server_key: "c66c3203ea77748e08a8db6eab845217"
server_key_iv: "ce8da4be6e02410654dcf8942acc3e3a"
server_key_version: "encryptionkey@0"
I20260807 18:28:16.931493 32621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 18:28:16.934849 32637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:16.936197 32621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:16.936371 32621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "af77369e02b14bbdb040d139dbc9e20e"
format_stamp: "Formatted at 2026-08-07 18:28:16 on dist-test-slave-1xrh"
server_key: "c66c3203ea77748e08a8db6eab845217"
server_key_iv: "ce8da4be6e02410654dcf8942acc3e3a"
server_key_version: "encryptionkey@0"
I20260807 18:28:16.936579 32621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:16.942173 32621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:16.943011 32621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:16.943295 32621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:16.944209 32621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:16.945566 32621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:16.945676 32621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:16.945796 32621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:16.945897 32621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:16.978161 32621 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:38357
I20260807 18:28:16.978220 32764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:38357 every 8 connection(s)
I20260807 18:28:16.979574 32621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:16.984004 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 32621
I20260807 18:28:16.984215 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:28:16.984869 27655 external_mini_cluster.cc:1428] Setting key ec461829c05d5ea42282f14481ae783d
I20260807 18:28:16.987546 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46181
--builtin_ntp_servers=127.27.1.212:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 18:28:16.988643 300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:16.982253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:33801 (local address 127.27.1.254:46181)
0807 18:28:16.982721 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:16.982727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:16.984053 (+ 1326us) server_negotiation.cc:407] Connection header received
0807 18:28:16.986012 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:16.986018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:16.986099 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:16.986226 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:16.987725 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:16.987731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:16.987736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:16.987777 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:16.987851 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:16.987868 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:16.988006 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:16.988219 (+ 213us) server_negotiation.cc:299] Negotiation successful
0807 18:28:16.988294 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":235,"threads_started":1}
I20260807 18:28:16.989692 302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:16.982841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:46181 (local address 127.27.1.193:33801)
0807 18:28:16.983880 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:16.983927 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:16.985731 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:16.986490 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:16.986498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:16.986592 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:16.987400 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:28:16.987404 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:16.987454 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:28:16.987546 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:16.987975 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:16.987987 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:16.987995 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 18:28:16.988322 (+ 327us) client_negotiation.cc:240] Negotiation successful
0807 18:28:16.988590 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":337,"threads_started":1}
I20260807 18:28:16.991391 32765 heartbeater.cc:347] Connected to a master server at 127.27.1.254:46181
I20260807 18:28:16.991693 32765 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:16.992318 32765 heartbeater.cc:511] Master 127.27.1.254:46181 requested a full tablet report, sending...
I20260807 18:28:16.994740 32559 ts_manager.cc:194] Registered new tserver with Master: af77369e02b14bbdb040d139dbc9e20e (127.27.1.193:38357)
W20260807 18:28:17.330153 304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:17.330513 304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:17.330683 304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:17.338862 304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:17.338970 304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:17.339083 304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:17.348440 304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:17.349673 304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:17.351501 304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:17.364328 313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:17.364418 310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:17.364408 311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:17.365363 304 server_base.cc:1034] running on GCE node
I20260807 18:28:17.366137 304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:17.367112 304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:17.368397 304 hybrid_clock.cc:648] HybridClock initialized: now 1786127297368343 us; error 47 us; skew 500 ppm
I20260807 18:28:17.368719 304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:28:17.371076 304 webserver.cc:460] Webserver started at http://127.27.1.194:38497/ using document root <none> and password file <none>
I20260807 18:28:17.371779 304 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:17.371886 304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:17.372159 304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:17.374992 304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "ea830297835e457581290a34cbb003b5"
format_stamp: "Formatted at 2026-08-07 18:28:17 on dist-test-slave-1xrh"
server_key: "7cde025b43715b03341304429531013b"
server_key_iv: "b55cf1efff71f02052ca92429405c29b"
server_key_version: "encryptionkey@0"
I20260807 18:28:17.376027 304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "ea830297835e457581290a34cbb003b5"
format_stamp: "Formatted at 2026-08-07 18:28:17 on dist-test-slave-1xrh"
server_key: "7cde025b43715b03341304429531013b"
server_key_iv: "b55cf1efff71f02052ca92429405c29b"
server_key_version: "encryptionkey@0"
I20260807 18:28:17.381914 304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260807 18:28:17.385311 319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:17.386639 304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 18:28:17.386905 304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "ea830297835e457581290a34cbb003b5"
format_stamp: "Formatted at 2026-08-07 18:28:17 on dist-test-slave-1xrh"
server_key: "7cde025b43715b03341304429531013b"
server_key_iv: "b55cf1efff71f02052ca92429405c29b"
server_key_version: "encryptionkey@0"
I20260807 18:28:17.387176 304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:17.392720 304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:17.393489 304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:17.393759 304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:17.394678 304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:17.396109 304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:17.396236 304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:17.396355 304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:17.396446 304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:17.429080 304 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:37079
I20260807 18:28:17.429162 441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:37079 every 8 connection(s)
I20260807 18:28:17.430388 304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:17.431977 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 304
I20260807 18:28:17.432144 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:28:17.432637 27655 external_mini_cluster.cc:1428] Setting key 56f42871695b71291e392e68bf1b2b11
I20260807 18:28:17.435622 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46181
--builtin_ntp_servers=127.27.1.212:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 18:28:17.439374 300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.433059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:50147 (local address 127.27.1.254:46181)
0807 18:28:17.433275 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:17.433282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:17.435293 (+ 2011us) server_negotiation.cc:407] Connection header received
0807 18:28:17.437054 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:17.437059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:17.437144 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:17.437255 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:17.438580 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:17.438586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:17.438591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:17.438610 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:17.438648 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:17.438659 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:17.438777 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:17.438978 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 18:28:17.439050 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 18:28:17.440394 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.433714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:46181 (local address 127.27.1.194:50147)
0807 18:28:17.435114 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:17.435194 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:17.436789 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:17.437492 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:17.437499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:17.437588 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:17.438288 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:28:17.438291 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:17.438326 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:28:17.438408 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:17.438753 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:17.438765 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:17.438772 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 18:28:17.439054 (+ 282us) client_negotiation.cc:240] Negotiation successful
0807 18:28:17.439327 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":678,"threads_started":1}
I20260807 18:28:17.441802 442 heartbeater.cc:347] Connected to a master server at 127.27.1.254:46181
I20260807 18:28:17.442083 442 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:17.442655 442 heartbeater.cc:511] Master 127.27.1.254:46181 requested a full tablet report, sending...
I20260807 18:28:17.444376 32559 ts_manager.cc:194] Registered new tserver with Master: ea830297835e457581290a34cbb003b5 (127.27.1.194:37079)
W20260807 18:28:17.764900 445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:17.765209 445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:17.765370 445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:17.772619 445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:17.772745 445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:17.772986 445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:17.781992 445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:17.783357 445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:17.785143 445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:17.799073 452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:17.799073 453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:17.799105 455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:17.799989 445 server_base.cc:1034] running on GCE node
I20260807 18:28:17.800585 445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:17.801458 445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:17.802687 445 hybrid_clock.cc:648] HybridClock initialized: now 1786127297802641 us; error 60 us; skew 500 ppm
I20260807 18:28:17.803016 445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:28:17.805511 445 webserver.cc:460] Webserver started at http://127.27.1.195:44077/ using document root <none> and password file <none>
I20260807 18:28:17.806208 445 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:17.806337 445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:17.806675 445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:17.809527 445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "e7641417429546ff823c625b993b302a"
format_stamp: "Formatted at 2026-08-07 18:28:17 on dist-test-slave-1xrh"
server_key: "193d94a0f8a371c48b082a660811fdb5"
server_key_iv: "642a6c74f452f277164a4b0d8b6dca1e"
server_key_version: "encryptionkey@0"
I20260807 18:28:17.810297 445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "e7641417429546ff823c625b993b302a"
format_stamp: "Formatted at 2026-08-07 18:28:17 on dist-test-slave-1xrh"
server_key: "193d94a0f8a371c48b082a660811fdb5"
server_key_iv: "642a6c74f452f277164a4b0d8b6dca1e"
server_key_version: "encryptionkey@0"
I20260807 18:28:17.815989 445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 18:28:17.820035 461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:17.821336 445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:28:17.821472 445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "e7641417429546ff823c625b993b302a"
format_stamp: "Formatted at 2026-08-07 18:28:17 on dist-test-slave-1xrh"
server_key: "193d94a0f8a371c48b082a660811fdb5"
server_key_iv: "642a6c74f452f277164a4b0d8b6dca1e"
server_key_version: "encryptionkey@0"
I20260807 18:28:17.821657 445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:17.827176 445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:17.828078 445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:17.828361 445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:17.829362 445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:17.830690 445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:17.830776 445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:17.830854 445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:17.830907 445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:17.863503 445 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:38441
I20260807 18:28:17.863565 583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:38441 every 8 connection(s)
I20260807 18:28:17.864810 445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:17.869026 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 445
I20260807 18:28:17.869194 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:28:17.869658 27655 external_mini_cluster.cc:1428] Setting key 3317be8ad2895beea122004c223bd79f
I20260807 18:28:17.873070 300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.867477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:46053 (local address 127.27.1.254:46181)
0807 18:28:17.867663 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:17.867668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:17.869057 (+ 1389us) server_negotiation.cc:407] Connection header received
0807 18:28:17.870849 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:17.870855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:17.870933 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:17.871036 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:17.872346 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:17.872351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:17.872356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:17.872374 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:17.872415 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:17.872425 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:17.872547 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:17.872731 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 18:28:17.872795 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 18:28:17.873915 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.868008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:46181 (local address 127.27.1.195:46053)
0807 18:28:17.868881 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:17.868916 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:17.870590 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:17.871289 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:17.871296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:17.871387 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:17.872055 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:28:17.872057 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:17.872093 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:28:17.872173 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:17.872514 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:17.872527 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:17.872536 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 18:28:17.872806 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 18:28:17.873033 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":271,"threads_started":1}
I20260807 18:28:17.875106 584 heartbeater.cc:347] Connected to a master server at 127.27.1.254:46181
I20260807 18:28:17.875385 584 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:17.875882 584 heartbeater.cc:511] Master 127.27.1.254:46181 requested a full tablet report, sending...
I20260807 18:28:17.877113 32559 ts_manager.cc:194] Registered new tserver with Master: e7641417429546ff823c625b993b302a (127.27.1.195:38441)
I20260807 18:28:17.884564 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:28:17.891749 300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.889100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.27.1.254:46181)
0807 18:28:17.889276 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:17.889281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:17.889770 (+ 489us) server_negotiation.cc:407] Connection header received
0807 18:28:17.889955 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:17.889960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:17.890031 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:17.890129 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:17.891041 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:17.891047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:17.891051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:17.891069 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:17.891119 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:17.891168 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:17.891277 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:17.891376 (+ 99us) server_negotiation.cc:299] Negotiation successful
0807 18:28:17.891435 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 18:28:17.895414 32559 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:28:17.901949 32559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:28:17.920262 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.913921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.27.1.194:37079)
0807 18:28:17.916030 (+ 2109us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:17.916041 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:17.916073 (+ 32us) server_negotiation.cc:407] Connection header received
0807 18:28:17.916149 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:17.916156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:17.916365 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:17.916545 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:17.918824 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:17.918833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:17.918847 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:17.918874 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:17.918934 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:17.918956 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:17.919689 (+ 733us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:17.919763 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 18:28:17.919866 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1904,"thread_start_us":1818,"threads_started":1}
I20260807 18:28:17.920327 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.913317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:37079 (local address 127.0.0.1:49092)
0807 18:28:17.915532 (+ 2215us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:17.915549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:17.915641 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:17.916546 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:17.916553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:17.916647 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:17.918512 (+ 1865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:28:17.918515 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:17.918553 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:28:17.918643 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:17.919107 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:17.919146 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:17.919156 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 18:28:17.919324 (+ 168us) client_negotiation.cc:240] Negotiation successful
0807 18:28:17.919412 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2069,"thread_start_us":4421,"threads_started":1}
I20260807 18:28:17.920917 604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.915583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.27.1.195:38441)
0807 18:28:17.916206 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:17.916213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:17.918233 (+ 2020us) server_negotiation.cc:407] Connection header received
0807 18:28:17.918294 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:17.918299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:17.918527 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:17.918685 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:17.919713 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:17.919721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:17.919733 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:17.919750 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:17.919803 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:17.919828 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:17.919983 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:17.920438 (+ 455us) server_negotiation.cc:299] Negotiation successful
0807 18:28:17.920551 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":389,"threads_started":1}
I20260807 18:28:17.921376 601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.915021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:38441 (local address 127.0.0.1:52700)
0807 18:28:17.917994 (+ 2973us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:17.918012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:17.918120 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:17.918710 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:17.918714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:17.918736 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:17.919452 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:28:17.919455 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:17.919471 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:28:17.919513 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:17.920045 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:17.920056 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:17.920062 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 18:28:17.920219 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 18:28:17.920275 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2849,"thread_start_us":369,"threads_started":1}
I20260807 18:28:17.922734 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.913251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.27.1.193:38357)
0807 18:28:17.918630 (+ 5379us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:17.918640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:17.918669 (+ 29us) server_negotiation.cc:407] Connection header received
0807 18:28:17.918733 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:17.918740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:17.918953 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:17.919808 (+ 855us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:17.921911 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:17.921918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:17.921931 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:17.921959 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:17.922008 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:17.922028 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:17.922161 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:17.922309 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 18:28:17.922397 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5234,"thread_start_us":4838,"threads_started":1}
I20260807 18:28:17.924243 372 tablet_service.cc:1467] Processing CreateTablet for tablet b206ed06c4804fdea971631be788d08b (DEFAULT_TABLE table=test-table [id=d62460d7487c43cba4fa3e85aa45eac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:17.924379 513 tablet_service.cc:1467] Processing CreateTablet for tablet b206ed06c4804fdea971631be788d08b (DEFAULT_TABLE table=test-table [id=d62460d7487c43cba4fa3e85aa45eac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:17.925633 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:17.913009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:38357 (local address 127.0.0.1:57074)
0807 18:28:17.913991 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:17.914032 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:17.914162 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:17.919449 (+ 5287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:17.919457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:17.919479 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:17.920840 (+ 1361us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:28:17.920843 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:17.920858 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:28:17.920899 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:17.922147 (+ 1248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:17.922154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:17.922158 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 18:28:17.922304 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 18:28:17.924832 (+ 2528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":590,"threads_started":1}
I20260807 18:28:17.925768 372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b206ed06c4804fdea971631be788d08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:17.925784 513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b206ed06c4804fdea971631be788d08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:17.929216 32694 tablet_service.cc:1467] Processing CreateTablet for tablet b206ed06c4804fdea971631be788d08b (DEFAULT_TABLE table=test-table [id=d62460d7487c43cba4fa3e85aa45eac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:17.930558 32694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b206ed06c4804fdea971631be788d08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:17.937763 605 tablet_bootstrap.cc:492] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: Bootstrap starting.
I20260807 18:28:17.941329 605 tablet_bootstrap.cc:654] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:17.942353 605 log.cc:824] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:17.946169 605 tablet_bootstrap.cc:492] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: No bootstrap required, opened a new log
I20260807 18:28:17.946308 608 tablet_bootstrap.cc:492] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e: Bootstrap starting.
I20260807 18:28:17.946455 605 ts_tablet_manager.cc:1397] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 18:28:17.947465 606 tablet_bootstrap.cc:492] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: Bootstrap starting.
I20260807 18:28:17.949235 608 tablet_bootstrap.cc:654] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:17.950238 608 log.cc:824] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:17.950842 606 tablet_bootstrap.cc:654] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:17.952102 606 log.cc:824] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:17.953267 608 tablet_bootstrap.cc:492] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e: No bootstrap required, opened a new log
I20260807 18:28:17.953533 608 ts_tablet_manager.cc:1397] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 18:28:17.955065 606 tablet_bootstrap.cc:492] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: No bootstrap required, opened a new log
I20260807 18:28:17.955677 606 ts_tablet_manager.cc:1397] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 18:28:17.956596 605 raft_consensus.cc:348] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af77369e02b14bbdb040d139dbc9e20e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38357 } } peers { permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 } } peers { permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 } }
I20260807 18:28:17.957046 605 raft_consensus.cc:374] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:17.957152 605 raft_consensus.cc:729] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea830297835e457581290a34cbb003b5, State: Initialized, Role: FOLLOWER
I20260807 18:28:17.957767 605 consensus_queue.cc:260] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af77369e02b14bbdb040d139dbc9e20e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38357 } } peers { permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 } } peers { permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 } }
I20260807 18:28:17.958835 442 heartbeater.cc:503] Master 127.27.1.254:46181 was elected leader, sending a full tablet report...
I20260807 18:28:17.959518 605 ts_tablet_manager.cc:1428] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260807 18:28:17.961571 608 raft_consensus.cc:348] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af77369e02b14bbdb040d139dbc9e20e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38357 } } peers { permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 } } peers { permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 } }
I20260807 18:28:17.961907 608 raft_consensus.cc:374] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:17.962010 608 raft_consensus.cc:729] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af77369e02b14bbdb040d139dbc9e20e, State: Initialized, Role: FOLLOWER
I20260807 18:28:17.962756 608 consensus_queue.cc:260] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af77369e02b14bbdb040d139dbc9e20e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38357 } } peers { permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 } } peers { permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 } }
I20260807 18:28:17.964001 32765 heartbeater.cc:503] Master 127.27.1.254:46181 was elected leader, sending a full tablet report...
I20260807 18:28:17.964643 608 ts_tablet_manager.cc:1428] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260807 18:28:17.965947 606 raft_consensus.cc:348] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af77369e02b14bbdb040d139dbc9e20e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38357 } } peers { permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 } } peers { permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 } }
I20260807 18:28:17.966406 606 raft_consensus.cc:374] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:17.966526 606 raft_consensus.cc:729] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7641417429546ff823c625b993b302a, State: Initialized, Role: FOLLOWER
I20260807 18:28:17.967164 606 consensus_queue.cc:260] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af77369e02b14bbdb040d139dbc9e20e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38357 } } peers { permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 } } peers { permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 } }
I20260807 18:28:17.968250 584 heartbeater.cc:503] Master 127.27.1.254:46181 was elected leader, sending a full tablet report...
I20260807 18:28:17.968767 606 ts_tablet_manager.cc:1428] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260807 18:28:17.981784 32767 tablet.cc:2369] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:28:18.074149 613 raft_consensus.cc:482] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:18.074492 613 raft_consensus.cc:504] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af77369e02b14bbdb040d139dbc9e20e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38357 } } peers { permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 } } peers { permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 } }
I20260807 18:28:18.076292 613 leader_election.cc:288] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea830297835e457581290a34cbb003b5 (127.27.1.194:37079), e7641417429546ff823c625b993b302a (127.27.1.195:38441)
I20260807 18:28:18.079793 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.076700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:36285 (local address 127.27.1.194:37079)
0807 18:28:18.076968 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:18.076975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:18.077820 (+ 845us) server_negotiation.cc:407] Connection header received
0807 18:28:18.077957 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:18.077961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:18.078008 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:18.078083 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:18.079078 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:18.079084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:18.079089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:18.079113 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:18.079188 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:18.079203 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:18.079340 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:18.079438 (+ 98us) server_negotiation.cc:299] Negotiation successful
0807 18:28:18.079514 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 18:28:18.079882 604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.077310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:37851 (local address 127.27.1.195:38441)
0807 18:28:18.077519 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:18.077525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:18.077547 (+ 22us) server_negotiation.cc:407] Connection header received
0807 18:28:18.077748 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:18.077756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:18.077828 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:18.077930 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:18.079276 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:18.079281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:18.079284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:18.079298 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:18.079328 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:18.079339 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:18.079477 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:18.079550 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 18:28:18.079616 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 18:28:18.080097 616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.076744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:37079 (local address 127.27.1.193:36285)
0807 18:28:18.077691 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:18.077715 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:18.077835 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:18.078141 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:18.078144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:18.078163 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:18.078861 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:28:18.078863 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:18.078878 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:28:18.078916 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:18.079299 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:18.079304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:18.079308 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 18:28:18.079459 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 18:28:18.079512 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":624,"threads_started":1}
I20260807 18:28:18.080152 615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.076609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:38441 (local address 127.27.1.193:37851)
0807 18:28:18.077420 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:18.077445 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:18.077590 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:18.078219 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:18.078222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:18.078240 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:18.079046 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:28:18.079049 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:18.079061 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:28:18.079095 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:18.079443 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:18.079447 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:18.079453 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 18:28:18.079606 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 18:28:18.079658 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":389,"threads_started":1}
I20260807 18:28:18.081188 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b206ed06c4804fdea971631be788d08b" candidate_uuid: "af77369e02b14bbdb040d139dbc9e20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7641417429546ff823c625b993b302a" is_pre_election: true
I20260807 18:28:18.081195 393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b206ed06c4804fdea971631be788d08b" candidate_uuid: "af77369e02b14bbdb040d139dbc9e20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea830297835e457581290a34cbb003b5" is_pre_election: true
I20260807 18:28:18.081681 535 raft_consensus.cc:2454] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af77369e02b14bbdb040d139dbc9e20e in term 0.
I20260807 18:28:18.081709 393 raft_consensus.cc:2454] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af77369e02b14bbdb040d139dbc9e20e in term 0.
I20260807 18:28:18.082342 32638 leader_election.cc:302] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: af77369e02b14bbdb040d139dbc9e20e, e7641417429546ff823c625b993b302a, ea830297835e457581290a34cbb003b5; no voters:
I20260807 18:28:18.082763 613 raft_consensus.cc:2781] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:28:18.082916 613 raft_consensus.cc:482] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:28:18.083024 613 raft_consensus.cc:3037] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:18.085345 613 raft_consensus.cc:504] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af77369e02b14bbdb040d139dbc9e20e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38357 } } peers { permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 } } peers { permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 } }
I20260807 18:28:18.085953 613 leader_election.cc:288] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [CANDIDATE]: Term 1 election: Requested vote from peers ea830297835e457581290a34cbb003b5 (127.27.1.194:37079), e7641417429546ff823c625b993b302a (127.27.1.195:38441)
I20260807 18:28:18.086465 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b206ed06c4804fdea971631be788d08b" candidate_uuid: "af77369e02b14bbdb040d139dbc9e20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7641417429546ff823c625b993b302a"
I20260807 18:28:18.086465 393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b206ed06c4804fdea971631be788d08b" candidate_uuid: "af77369e02b14bbdb040d139dbc9e20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea830297835e457581290a34cbb003b5"
I20260807 18:28:18.086683 535 raft_consensus.cc:3037] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:18.086683 393 raft_consensus.cc:3037] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:18.090107 393 raft_consensus.cc:2454] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af77369e02b14bbdb040d139dbc9e20e in term 1.
I20260807 18:28:18.090130 535 raft_consensus.cc:2454] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af77369e02b14bbdb040d139dbc9e20e in term 1.
I20260807 18:28:18.090533 32638 leader_election.cc:302] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: af77369e02b14bbdb040d139dbc9e20e, e7641417429546ff823c625b993b302a, ea830297835e457581290a34cbb003b5; no voters:
I20260807 18:28:18.090818 613 raft_consensus.cc:2781] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:18.091318 613 raft_consensus.cc:686] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [term 1 LEADER]: Becoming Leader. State: Replica: af77369e02b14bbdb040d139dbc9e20e, State: Running, Role: LEADER
I20260807 18:28:18.091892 613 consensus_queue.cc:237] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af77369e02b14bbdb040d139dbc9e20e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38357 } } peers { permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 } } peers { permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 } }
I20260807 18:28:18.096464 32558 catalog_manager.cc:5495] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e reported cstate change: term changed from 0 to 1, leader changed from <none> to af77369e02b14bbdb040d139dbc9e20e (127.27.1.193). New cstate: current_term: 1 leader_uuid: "af77369e02b14bbdb040d139dbc9e20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af77369e02b14bbdb040d139dbc9e20e" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 } health_report { overall_health: UNKNOWN } } }
I20260807 18:28:18.110558 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.108610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.27.1.193:38357)
0807 18:28:18.108817 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:18.108822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:18.108843 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:28:18.109040 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:18.109043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:18.109117 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:18.109196 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:18.109974 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:18.109978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:18.109981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:18.109996 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:18.110025 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:18.110036 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:18.110125 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:18.110212 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 18:28:18.110257 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260807 18:28:18.116436 585 tablet.cc:2369] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:28:18.118057 393 raft_consensus.cc:1261] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5 [term 1 FOLLOWER]: Refusing update from remote peer af77369e02b14bbdb040d139dbc9e20e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:18.118083 535 raft_consensus.cc:1261] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a [term 1 FOLLOWER]: Refusing update from remote peer af77369e02b14bbdb040d139dbc9e20e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:18.118950 617 consensus_queue.cc:1035] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7641417429546ff823c625b993b302a" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:18.119350 613 consensus_queue.cc:1035] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea830297835e457581290a34cbb003b5" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 37079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:18.139436 623 mvcc.cc:205] Tried to move back new op lower bound from 7315977413082472448 to 7315977412988157952. Current Snapshot: MvccSnapshot[applied={T|T < 7315977413082472448}]
I20260807 18:28:18.152089 32558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:18.152380 32558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:18.156415 32558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7499b020d72942ba9f47298c01880821: Sending DeleteTablet for 3 replicas of tablet b206ed06c4804fdea971631be788d08b
I20260807 18:28:18.157610 372 tablet_service.cc:1514] Processing DeleteTablet for tablet b206ed06c4804fdea971631be788d08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:18 UTC) from {username='slave'} at 127.0.0.1:49092
I20260807 18:28:18.157645 32694 tablet_service.cc:1514] Processing DeleteTablet for tablet b206ed06c4804fdea971631be788d08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:18 UTC) from {username='slave'} at 127.0.0.1:57074
I20260807 18:28:18.157684 513 tablet_service.cc:1514] Processing DeleteTablet for tablet b206ed06c4804fdea971631be788d08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:18 UTC) from {username='slave'} at 127.0.0.1:52700
I20260807 18:28:18.159317 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 32621
I20260807 18:28:18.186753 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.160614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57096 (local address 127.27.1.193:38357)
0807 18:28:18.160802 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:18.160810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:18.181371 (+ 20561us) server_negotiation.cc:407] Connection header received
0807 18:28:18.182068 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:18.182083 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:18.182194 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:18.182291 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:18.185635 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:18.185644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:18.185652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:18.185677 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:18.185718 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:18.185734 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:18.185837 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:18.186206 (+ 369us) server_negotiation.cc:299] Negotiation successful
0807 18:28:18.186297 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 18:28:18.188395 32760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:18.189347 634 tablet_replica.cc:330] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: stopping tablet replica
I20260807 18:28:18.189968 634 raft_consensus.cc:2229] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:18.190711 634 raft_consensus.cc:2258] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:18.193900 633 tablet_replica.cc:330] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: stopping tablet replica
I20260807 18:28:18.194280 634 ts_tablet_manager.cc:1905] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:18.194391 633 raft_consensus.cc:2229] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:18.194941 633 raft_consensus.cc:2258] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:18.196905 633 ts_tablet_manager.cc:1905] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:18.201391 634 ts_tablet_manager.cc:1918] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:18.201556 634 log.cc:1196] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/b206ed06c4804fdea971631be788d08b
I20260807 18:28:18.202071 634 ts_tablet_manager.cc:1939] T b206ed06c4804fdea971631be788d08b P ea830297835e457581290a34cbb003b5: Deleting consensus metadata
I20260807 18:28:18.203862 633 ts_tablet_manager.cc:1918] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:18.204036 633 log.cc:1196] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/b206ed06c4804fdea971631be788d08b
I20260807 18:28:18.204056 32544 catalog_manager.cc:4841] TS ea830297835e457581290a34cbb003b5 (127.27.1.194:37079): tablet b206ed06c4804fdea971631be788d08b (table test-table [id=d62460d7487c43cba4fa3e85aa45eac5]) successfully deleted
I20260807 18:28:18.204615 633 ts_tablet_manager.cc:1939] T b206ed06c4804fdea971631be788d08b P e7641417429546ff823c625b993b302a: Deleting consensus metadata
I20260807 18:28:18.206085 32540 catalog_manager.cc:4841] TS e7641417429546ff823c625b993b302a (127.27.1.195:38441): tablet b206ed06c4804fdea971631be788d08b (table test-table [id=d62460d7487c43cba4fa3e85aa45eac5]) successfully deleted
I20260807 18:28:18.289280 632 tablet_replica.cc:330] T b206ed06c4804fdea971631be788d08b P af77369e02b14bbdb040d139dbc9e20e: stopping tablet replica
I20260807 18:28:18.289561 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 32621
W20260807 18:28:18.309974 32542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.1.193:38357 (error 108)
I20260807 18:28:18.310714 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 304
I20260807 18:28:18.312134 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.311424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:38357 (local address 127.0.0.1:57112)
0807 18:28:18.311795 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:18.311966 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:38357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
W20260807 18:28:18.312399 32542 catalog_manager.cc:4584] TS af77369e02b14bbdb040d139dbc9e20e (127.27.1.193:38357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b206ed06c4804fdea971631be788d08b: Network error: Client connection negotiation failed: client connection to 127.27.1.193:38357: connect: Connection refused (error 111)
I20260807 18:28:18.313786 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.311612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.27.1.194:37079)
0807 18:28:18.311781 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:18.311786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:18.311804 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:28:18.311942 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:18.311946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:18.312033 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:18.312116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:18.313159 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:18.313170 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:18.313173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:18.313189 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:18.313221 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:18.313233 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:18.313375 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:18.313468 (+ 93us) server_negotiation.cc:299] Negotiation successful
0807 18:28:18.313527 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:28:18.314498 436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:18.366195 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.365694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:38357 (local address 127.0.0.1:57126)
0807 18:28:18.365917 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:18.366032 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:38357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 18:28:18.367326 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.366912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:38357 (local address 127.0.0.1:57140)
0807 18:28:18.367073 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:18.367195 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:38357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 18:28:18.402592 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 304
I20260807 18:28:18.423808 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 445
I20260807 18:28:18.426879 604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.424700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.27.1.195:38441)
0807 18:28:18.424892 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:18.424898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:18.424919 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:28:18.425085 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:18.425090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:18.425172 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:18.425278 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:18.426228 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:18.426234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:18.426239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:18.426256 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:18.426294 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:18.426306 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:18.426447 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:18.426533 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 18:28:18.426595 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 18:28:18.427513 578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:18.427646 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.427224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:38357 (local address 127.0.0.1:57146)
0807 18:28:18.427400 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:18.427487 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:38357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 18:28:18.428937 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.428536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:38357 (local address 127.0.0.1:57152)
0807 18:28:18.428687 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:18.428779 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:38357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260807 18:28:18.509647 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.509104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:38357 (local address 127.0.0.1:57160)
0807 18:28:18.509344 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:18.509457 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:38357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260807 18:28:18.510789 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:18.510396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:38357 (local address 127.0.0.1:57166)
0807 18:28:18.510595 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:18.510668 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:38357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260807 18:28:18.516206 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 445
I20260807 18:28:18.535704 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 32525
I20260807 18:28:18.536810 32591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:18.642324 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 32525
2026-08-07T18:28:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2681 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 18:28:18.688519 27655 mini_oidc.cc:152] Starting JWKS server
I20260807 18:28:18.691381 27655 webserver.cc:460] Webserver started at https://127.0.0.1:43103/ using document root <none> and password file <none>
I20260807 18:28:18.691733 27655 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 18:28:18.692399 27655 webserver.cc:460] Webserver started at http://127.0.0.1:40499/ using document root <none> and password file <none>
2026-08-07T18:28:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:18Z Disabled control of system clock
I20260807 18:28:18.724282 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34185
--webserver_interface=127.27.1.254
--webserver_port=0
--builtin_ntp_servers=127.27.1.212:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:34185
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43103/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/testchainca.cert with env {}
W20260807 18:28:19.051646 647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:19.051965 647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:19.052132 647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:19.059301 647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:19.059425 647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:19.059505 647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:19.059607 647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:19.059685 647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 18:28:19.059772 647 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 18:28:19.059827 647 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43103/jwks.json
I20260807 18:28:19.070086 647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:34185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43103/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:19.071522 647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:19.072889 647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:19.084910 652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:19.084944 655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:19.085007 653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:19.085932 647 server_base.cc:1034] running on GCE node
I20260807 18:28:19.086730 647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:19.088140 647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:19.089370 647 hybrid_clock.cc:648] HybridClock initialized: now 1786127299089330 us; error 51 us; skew 500 ppm
I20260807 18:28:19.089711 647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:28:19.092059 647 webserver.cc:460] Webserver started at http://127.27.1.254:35107/ using document root <none> and password file <none>
I20260807 18:28:19.092774 647 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:19.092904 647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:19.093217 647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:19.096575 647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "4ffb9d88d715457580aeebfc8165f814"
format_stamp: "Formatted at 2026-08-07 18:28:19 on dist-test-slave-1xrh"
server_key: "7722202dcf0ba78fdf04bcc84fd30271"
server_key_iv: "587c86118843a2a53d3b471c1dd4e3f8"
server_key_version: "encryptionkey@0"
I20260807 18:28:19.097373 647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "4ffb9d88d715457580aeebfc8165f814"
format_stamp: "Formatted at 2026-08-07 18:28:19 on dist-test-slave-1xrh"
server_key: "7722202dcf0ba78fdf04bcc84fd30271"
server_key_iv: "587c86118843a2a53d3b471c1dd4e3f8"
server_key_version: "encryptionkey@0"
I20260807 18:28:19.102387 647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 18:28:19.105751 661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:19.106950 647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:28:19.107172 647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "4ffb9d88d715457580aeebfc8165f814"
format_stamp: "Formatted at 2026-08-07 18:28:19 on dist-test-slave-1xrh"
server_key: "7722202dcf0ba78fdf04bcc84fd30271"
server_key_iv: "587c86118843a2a53d3b471c1dd4e3f8"
server_key_version: "encryptionkey@0"
I20260807 18:28:19.107402 647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:19.123660 647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:19.124445 647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:19.124748 647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:19.143326 647 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:34185
I20260807 18:28:19.143332 717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:34185 every 8 connection(s)
I20260807 18:28:19.144670 647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:19.148518 718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:19.153111 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 647
I20260807 18:28:19.153326 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:19.153955 27655 external_mini_cluster.cc:1428] Setting key 5d080a07e5218da5f52e96e265f9285b
I20260807 18:28:19.160132 718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814: Bootstrap starting.
I20260807 18:28:19.163468 718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:19.164690 718 log.cc:824] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:19.167954 718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814: No bootstrap required, opened a new log
I20260807 18:28:19.167964 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:19.155906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.27.1.254:34185)
0807 18:28:19.156983 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:19.156996 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:19.157040 (+ 44us) server_negotiation.cc:407] Connection header received
0807 18:28:19.158748 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:19.158769 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:19.159275 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:19.159627 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:19.160204 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:19.161533 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:19.162862 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:19.163629 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:19.163854 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:19.165228 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:19.165243 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:19.165270 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:19.166102 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:19.166208 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:19.166234 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:19.166408 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:19.166641 (+ 233us) server_negotiation.cc:299] Negotiation successful
0807 18:28:19.166966 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":394,"threads_started":1}
I20260807 18:28:19.171941 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:28:19.176887 718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffb9d88d715457580aeebfc8165f814" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34185 } }
I20260807 18:28:19.177277 718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:19.177414 718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ffb9d88d715457580aeebfc8165f814, State: Initialized, Role: FOLLOWER
I20260807 18:28:19.177986 718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffb9d88d715457580aeebfc8165f814" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34185 } }
I20260807 18:28:19.178236 718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:19.178393 718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:19.178586 718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:19.181154 718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffb9d88d715457580aeebfc8165f814" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34185 } }
I20260807 18:28:19.181759 718 leader_election.cc:302] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ffb9d88d715457580aeebfc8165f814; no voters:
I20260807 18:28:19.182363 718 leader_election.cc:288] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:19.182487 724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:19.183280 724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [term 1 LEADER]: Becoming Leader. State: Replica: 4ffb9d88d715457580aeebfc8165f814, State: Running, Role: LEADER
I20260807 18:28:19.183772 724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffb9d88d715457580aeebfc8165f814" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34185 } }
I20260807 18:28:19.184576 718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:19.186920 726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ffb9d88d715457580aeebfc8165f814. Latest consensus state: current_term: 1 leader_uuid: "4ffb9d88d715457580aeebfc8165f814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffb9d88d715457580aeebfc8165f814" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34185 } } }
I20260807 18:28:19.187542 726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:19.187369 725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ffb9d88d715457580aeebfc8165f814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffb9d88d715457580aeebfc8165f814" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34185 } } }
I20260807 18:28:19.187842 725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:19.189226 729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:19.195566 729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:19.204732 729 catalog_manager.cc:1347] Generated new cluster ID: 2315a59be7f04bc6bff87cc74b2c64fc
I20260807 18:28:19.204862 729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:19.223443 729 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:19.224720 729 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:19.234531 729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ffb9d88d715457580aeebfc8165f814: Generated new TSK 0
I20260807 18:28:19.235451 729 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 18:28:19.274720 750 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 18:28:19.274960 750 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 18:28:19.277318 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:19.254191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.27.1.254:34185)
0807 18:28:19.254418 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:19.254425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:19.254818 (+ 393us) server_negotiation.cc:407] Connection header received
0807 18:28:19.255199 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:19.255206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:19.255309 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:19.255498 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0807 18:28:19.255996 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:19.257144 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:19.258528 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:19.259253 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:19.259395 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:19.259855 (+ 460us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 18:28:19.276615 (+ 16760us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 18:28:19.277020 (+ 405us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37068: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":80}
W20260807 18:28:19.277446 721 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37068: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 18:28:19.278774 749 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:19.254042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34185 (local address 127.0.0.1:37068)
0807 18:28:19.254701 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:19.254721 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:19.254885 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:19.255518 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:19.255522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:19.255532 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0807 18:28:19.255852 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:19.255862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:19.257283 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:19.257287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:19.258389 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:19.258400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:19.259420 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:19.259424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:19.259540 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:19.259595 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 18:28:19.276900 (+ 17305us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 18:28:19.277152 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.1.254:34185: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":343,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.1.254:34185: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 18:28:19.280623 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 647
I20260807 18:28:19.281769 713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:19.381562 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 647
2026-08-07T18:28:19Z chronyd exiting
I20260807 18:28:19.435672 27655 test_util.cc:175] -----------------------------------------------
I20260807 18:28:19.435812 27655 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786127255008662-27655-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (752 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 18:28:19.440402 27655 mini_oidc.cc:152] Starting JWKS server
I20260807 18:28:19.442257 27655 webserver.cc:460] Webserver started at https://127.0.0.1:36139/ using document root <none> and password file <none>
I20260807 18:28:19.442508 27655 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 18:28:19.443225 27655 webserver.cc:460] Webserver started at http://127.0.0.1:36639/ using document root <none> and password file <none>
2026-08-07T18:28:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:19Z Disabled control of system clock
I20260807 18:28:19.475286 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:46615
--webserver_interface=127.27.1.254
--webserver_port=0
--builtin_ntp_servers=127.27.1.212:35835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:46615
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36139/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/testchainca.cert with env {}
W20260807 18:28:19.801863 757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:19.802189 757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:19.802341 757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:19.809335 757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:19.809479 757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:19.809567 757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:19.809685 757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:19.809764 757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 18:28:19.809852 757 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 18:28:19.809932 757 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36139/jwks.json
I20260807 18:28:19.819628 757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:46615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36139/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:19.820921 757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:19.822495 757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:19.834996 766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:19.835122 763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:19.835402 764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:19.835745 757 server_base.cc:1034] running on GCE node
I20260807 18:28:19.836582 757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:19.837952 757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:19.839205 757 hybrid_clock.cc:648] HybridClock initialized: now 1786127299839143 us; error 52 us; skew 500 ppm
I20260807 18:28:19.839547 757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:28:19.841810 757 webserver.cc:460] Webserver started at http://127.27.1.254:36963/ using document root <none> and password file <none>
I20260807 18:28:19.842489 757 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:19.842628 757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:19.842967 757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:19.845942 757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "ac7c99a427ca4c9a856b88794eb81738"
format_stamp: "Formatted at 2026-08-07 18:28:19 on dist-test-slave-1xrh"
server_key: "4e69d66dfe5a552baebb2f09b6125e0f"
server_key_iv: "00694fc679c67eed03288f30d28ae9b1"
server_key_version: "encryptionkey@0"
I20260807 18:28:19.846719 757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "ac7c99a427ca4c9a856b88794eb81738"
format_stamp: "Formatted at 2026-08-07 18:28:19 on dist-test-slave-1xrh"
server_key: "4e69d66dfe5a552baebb2f09b6125e0f"
server_key_iv: "00694fc679c67eed03288f30d28ae9b1"
server_key_version: "encryptionkey@0"
I20260807 18:28:19.851631 757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260807 18:28:19.855222 774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:19.856477 757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:28:19.856660 757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "ac7c99a427ca4c9a856b88794eb81738"
format_stamp: "Formatted at 2026-08-07 18:28:19 on dist-test-slave-1xrh"
server_key: "4e69d66dfe5a552baebb2f09b6125e0f"
server_key_iv: "00694fc679c67eed03288f30d28ae9b1"
server_key_version: "encryptionkey@0"
I20260807 18:28:19.856879 757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:19.876760 757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:19.877560 757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:19.877883 757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:19.896248 757 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:46615
I20260807 18:28:19.896286 829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:46615 every 8 connection(s)
I20260807 18:28:19.897529 757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:19.901438 830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:19.904350 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 757
I20260807 18:28:19.904547 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:19.905485 27655 external_mini_cluster.cc:1428] Setting key 6443fc47d4707f01849105239c387425
I20260807 18:28:19.914381 830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738: Bootstrap starting.
I20260807 18:28:19.917685 830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:19.918568 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:19.907621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57268 (local address 127.27.1.254:46615)
0807 18:28:19.908641 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:19.908651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:19.908691 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:28:19.910279 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:19.910300 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:19.910720 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:19.911104 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:19.911668 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:19.912944 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:19.914170 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:19.914884 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:19.915072 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:19.916327 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:19.916335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:19.916347 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:19.916867 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:19.916955 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:19.916975 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:19.917105 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:19.917325 (+ 220us) server_negotiation.cc:299] Negotiation successful
0807 18:28:19.917617 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":391,"threads_started":1}
I20260807 18:28:19.918915 830 log.cc:824] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:19.922165 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:28:19.922188 830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738: No bootstrap required, opened a new log
I20260807 18:28:19.930651 830 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7c99a427ca4c9a856b88794eb81738" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46615 } }
I20260807 18:28:19.931041 830 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:19.931182 830 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac7c99a427ca4c9a856b88794eb81738, State: Initialized, Role: FOLLOWER
I20260807 18:28:19.931839 830 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7c99a427ca4c9a856b88794eb81738" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46615 } }
I20260807 18:28:19.932091 830 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:19.932265 830 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:19.932461 830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:19.934868 830 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7c99a427ca4c9a856b88794eb81738" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46615 } }
I20260807 18:28:19.935469 830 leader_election.cc:302] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac7c99a427ca4c9a856b88794eb81738; no voters:
I20260807 18:28:19.936098 830 leader_election.cc:288] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:19.936219 836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:19.936965 836 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [term 1 LEADER]: Becoming Leader. State: Replica: ac7c99a427ca4c9a856b88794eb81738, State: Running, Role: LEADER
I20260807 18:28:19.937475 836 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7c99a427ca4c9a856b88794eb81738" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46615 } }
I20260807 18:28:19.938287 830 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:19.940620 838 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac7c99a427ca4c9a856b88794eb81738. Latest consensus state: current_term: 1 leader_uuid: "ac7c99a427ca4c9a856b88794eb81738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7c99a427ca4c9a856b88794eb81738" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46615 } } }
I20260807 18:28:19.940507 837 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac7c99a427ca4c9a856b88794eb81738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7c99a427ca4c9a856b88794eb81738" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46615 } } }
I20260807 18:28:19.940981 838 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:19.941102 837 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:19.943255 843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:19.948524 843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:19.957654 843 catalog_manager.cc:1347] Generated new cluster ID: 20f0bf9f42fb46fabe925324a73f878b
I20260807 18:28:19.957767 843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:19.977262 843 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:19.978772 843 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:19.984979 843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac7c99a427ca4c9a856b88794eb81738: Generated new TSK 0
I20260807 18:28:19.985790 843 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 18:28:20.019362 863 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 18:28:20.019510 863 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 18:28:20.020769 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:20.000393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.27.1.254:46615)
0807 18:28:20.000616 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:20.000624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:20.001054 (+ 430us) server_negotiation.cc:407] Connection header received
0807 18:28:20.001272 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:20.001278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:20.001382 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:20.001576 (+ 194us) server_negotiation.cc:226] Negotiated authn=JWT
0807 18:28:20.002064 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:20.003224 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:20.004687 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:20.005379 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:20.005517 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:20.006014 (+ 497us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 18:28:20.020088 (+ 14074us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 18:28:20.020475 (+ 387us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 18:28:20.020895 862 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:20.000258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:46615 (local address 127.0.0.1:57274)
0807 18:28:20.000948 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:20.000965 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:20.001092 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:20.001594 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:20.001598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:20.001609 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0807 18:28:20.001887 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:20.001895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:20.003404 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:20.003409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:20.004527 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:20.004540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:20.005554 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:20.005559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:20.005701 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:20.005737 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 18:28:20.020344 (+ 14607us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 18:28:20.020561 (+ 217us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.1.254:46615: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":377,"threads_started":1}
W20260807 18:28:20.020907 834 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260807 18:28:20.022039 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 757
I20260807 18:28:20.023419 825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:20.114869 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 757
2026-08-07T18:28:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (733 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 18:28:20.173103 27655 mini_oidc.cc:152] Starting JWKS server
I20260807 18:28:20.175010 27655 webserver.cc:460] Webserver started at https://127.0.0.1:33241/ using document root <none> and password file <none>
I20260807 18:28:20.175297 27655 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 18:28:20.175907 27655 webserver.cc:460] Webserver started at http://127.0.0.1:37251/ using document root <none> and password file <none>
2026-08-07T18:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:20Z Disabled control of system clock
I20260807 18:28:20.207981 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:45653
--webserver_interface=127.27.1.254
--webserver_port=0
--builtin_ntp_servers=127.27.1.212:37791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:45653
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33241/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 18:28:20.532907 870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:20.533247 870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:20.533416 870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:20.540934 870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:20.541075 870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:20.541165 870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:20.541271 870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:20.541356 870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 18:28:20.541445 870 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 18:28:20.541522 870 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33241/jwks.json
I20260807 18:28:20.550585 870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:45653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:45653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33241/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:20.551973 870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:20.553764 870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:20.565676 877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:20.565742 878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:20.565717 880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:20.566488 870 server_base.cc:1034] running on GCE node
I20260807 18:28:20.567332 870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:20.568759 870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:20.569985 870 hybrid_clock.cc:648] HybridClock initialized: now 1786127300569954 us; error 33 us; skew 500 ppm
I20260807 18:28:20.570318 870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:28:20.572620 870 webserver.cc:460] Webserver started at http://127.27.1.254:32947/ using document root <none> and password file <none>
I20260807 18:28:20.573292 870 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:20.573436 870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:20.573767 870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:20.576766 870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "2a8730ead1664f48af63ab8cd0928330"
format_stamp: "Formatted at 2026-08-07 18:28:20 on dist-test-slave-1xrh"
server_key: "de94f83bb22638bb6027a21e47cc3f2d"
server_key_iv: "006bbef476f9befe56a267af818cbec5"
server_key_version: "encryptionkey@0"
I20260807 18:28:20.577554 870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "2a8730ead1664f48af63ab8cd0928330"
format_stamp: "Formatted at 2026-08-07 18:28:20 on dist-test-slave-1xrh"
server_key: "de94f83bb22638bb6027a21e47cc3f2d"
server_key_iv: "006bbef476f9befe56a267af818cbec5"
server_key_version: "encryptionkey@0"
I20260807 18:28:20.582494 870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:28:20.585800 886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:20.587205 870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 18:28:20.587402 870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "2a8730ead1664f48af63ab8cd0928330"
format_stamp: "Formatted at 2026-08-07 18:28:20 on dist-test-slave-1xrh"
server_key: "de94f83bb22638bb6027a21e47cc3f2d"
server_key_iv: "006bbef476f9befe56a267af818cbec5"
server_key_version: "encryptionkey@0"
I20260807 18:28:20.587626 870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:20.602504 870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:20.603312 870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:20.603628 870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:20.621869 870 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:45653
I20260807 18:28:20.621907 945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:45653 every 8 connection(s)
I20260807 18:28:20.623220 870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:20.626910 946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:20.627390 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 870
I20260807 18:28:20.627614 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:20.628445 27655 external_mini_cluster.cc:1428] Setting key f4bed211980c12914a0d88346de61507
I20260807 18:28:20.640772 946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330: Bootstrap starting.
I20260807 18:28:20.642247 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:20.630336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.27.1.254:45653)
0807 18:28:20.631419 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:20.631429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:20.631469 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:28:20.633252 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:20.633272 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:20.633684 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:20.633997 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:20.634603 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:20.635900 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:20.637807 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:20.638601 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:20.638724 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:20.640019 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:20.640027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:20.640038 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:20.640530 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:20.640602 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:20.640626 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:20.640709 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:20.641003 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 18:28:20.641315 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":379,"threads_started":1}
I20260807 18:28:20.644052 946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:20.645236 946 log.cc:824] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:20.645828 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:28:20.647997 946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330: No bootstrap required, opened a new log
I20260807 18:28:20.655479 946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8730ead1664f48af63ab8cd0928330" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45653 } }
I20260807 18:28:20.655831 946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:20.655941 946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8730ead1664f48af63ab8cd0928330, State: Initialized, Role: FOLLOWER
I20260807 18:28:20.656526 946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8730ead1664f48af63ab8cd0928330" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45653 } }
I20260807 18:28:20.656769 946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:20.656924 946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:20.657126 946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:20.659276 946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8730ead1664f48af63ab8cd0928330" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45653 } }
I20260807 18:28:20.659744 946 leader_election.cc:302] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a8730ead1664f48af63ab8cd0928330; no voters:
I20260807 18:28:20.660320 946 leader_election.cc:288] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:20.660491 951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:20.661199 951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [term 1 LEADER]: Becoming Leader. State: Replica: 2a8730ead1664f48af63ab8cd0928330, State: Running, Role: LEADER
I20260807 18:28:20.661773 951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8730ead1664f48af63ab8cd0928330" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45653 } }
I20260807 18:28:20.662328 946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:20.664223 952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a8730ead1664f48af63ab8cd0928330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8730ead1664f48af63ab8cd0928330" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45653 } } }
I20260807 18:28:20.664561 952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:20.665123 953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a8730ead1664f48af63ab8cd0928330. Latest consensus state: current_term: 1 leader_uuid: "2a8730ead1664f48af63ab8cd0928330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8730ead1664f48af63ab8cd0928330" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45653 } } }
I20260807 18:28:20.665421 956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:20.665526 953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:20.671422 956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:20.680228 956 catalog_manager.cc:1347] Generated new cluster ID: df386e5328c642898faeefad84531076
I20260807 18:28:20.680341 956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:20.699402 956 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:20.701153 956 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:20.712848 956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a8730ead1664f48af63ab8cd0928330: Generated new TSK 0
I20260807 18:28:20.713804 956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:28:20.737824 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:20.729106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.27.1.254:45653)
0807 18:28:20.729342 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:20.729350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:20.729721 (+ 371us) server_negotiation.cc:407] Connection header received
0807 18:28:20.729964 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:20.729988 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:20.730090 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:20.730224 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0807 18:28:20.730777 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:20.731557 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:20.733530 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:20.733986 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:20.734083 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:20.734711 (+ 628us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 18:28:20.737188 (+ 2477us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33241/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 18:28:20.737522 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33241/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":69}
W20260807 18:28:20.737948 949 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33241/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260807 18:28:20.739984 977 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:20.729012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:45653 (local address 127.0.0.1:56238)
0807 18:28:20.729614 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:20.729634 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:20.729781 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:20.730268 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:20.730274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:20.730286 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0807 18:28:20.730615 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:20.730630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:20.731699 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:20.731705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:20.733385 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:20.733399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:20.734193 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:20.734199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:20.734386 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:20.734433 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 18:28:20.737399 (+ 2966us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33241/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 18:28:20.739562 (+ 2163us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.1.254:45653: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33241/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":1871,"thread_start_us":309,"threads_started":1}
I20260807 18:28:20.741044 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 870
I20260807 18:28:20.742870 941 generic_service.cc:196] Checking for leaks (request via RPC)
E20260807 18:28:20.750707 978 webserver.cc:533] Webserver: sslize failed:
E20260807 18:28:20.750830 978 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 18:28:20.836648 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 870
2026-08-07T18:28:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (721 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T18:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:20Z Disabled control of system clock
I20260807 18:28:20.927742 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34197
--webserver_interface=127.27.1.254
--webserver_port=0
--builtin_ntp_servers=127.27.1.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:34197
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/testchainca.cert with env {}
W20260807 18:28:21.286286 983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:21.286626 983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:21.286734 983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:21.293823 983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:21.293918 983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:21.293955 983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:21.294004 983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:21.294051 983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 18:28:21.294111 983 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 18:28:21.294152 983 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 18:28:21.305073 983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:34197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:21.306342 983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:21.307724 983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:21.321882 993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:21.321832 991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:21.321849 990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:21.323077 983 server_base.cc:1034] running on GCE node
I20260807 18:28:21.324011 983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:21.325484 983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:21.326802 983 hybrid_clock.cc:648] HybridClock initialized: now 1786127301326747 us; error 36 us; skew 500 ppm
I20260807 18:28:21.327323 983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:28:21.329628 983 webserver.cc:460] Webserver started at http://127.27.1.254:37873/ using document root <none> and password file <none>
I20260807 18:28:21.330344 983 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:21.330480 983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:21.330806 983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:21.333739 983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "80f5a75a628745ea9aab5086756d60ab"
format_stamp: "Formatted at 2026-08-07 18:28:21 on dist-test-slave-1xrh"
server_key: "1ec5a18b45f4a2a2a6ffef4aac641ca8"
server_key_iv: "e1a73ed85c0b914410e528a1e1fed46c"
server_key_version: "encryptionkey@0"
I20260807 18:28:21.334563 983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "80f5a75a628745ea9aab5086756d60ab"
format_stamp: "Formatted at 2026-08-07 18:28:21 on dist-test-slave-1xrh"
server_key: "1ec5a18b45f4a2a2a6ffef4aac641ca8"
server_key_iv: "e1a73ed85c0b914410e528a1e1fed46c"
server_key_version: "encryptionkey@0"
I20260807 18:28:21.339926 983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 18:28:21.343516 999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:21.344944 983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:28:21.345142 983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "80f5a75a628745ea9aab5086756d60ab"
format_stamp: "Formatted at 2026-08-07 18:28:21 on dist-test-slave-1xrh"
server_key: "1ec5a18b45f4a2a2a6ffef4aac641ca8"
server_key_iv: "e1a73ed85c0b914410e528a1e1fed46c"
server_key_version: "encryptionkey@0"
I20260807 18:28:21.345368 983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:21.365962 983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:21.366827 983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:21.367182 983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:21.386305 983 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:34197
I20260807 18:28:21.386346 1055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:34197 every 8 connection(s)
I20260807 18:28:21.387742 983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:21.388777 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 983
I20260807 18:28:21.388984 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:21.389602 27655 external_mini_cluster.cc:1428] Setting key 34ef8ba16fde88888cd5c560864e3682
I20260807 18:28:21.392023 1056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:21.403319 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:21.391431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.27.1.254:34197)
0807 18:28:21.392463 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:21.392474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:21.392516 (+ 42us) server_negotiation.cc:407] Connection header received
0807 18:28:21.394044 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:21.394065 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:21.394484 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:21.394843 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:21.395450 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:21.396819 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:21.398737 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:21.399453 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:21.399583 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:21.400931 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:21.400942 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:21.400955 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:28:21.401499 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:21.401575 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:28:21.401609 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:21.401723 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:21.401990 (+ 267us) server_negotiation.cc:299] Negotiation successful
0807 18:28:21.402341 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":385,"threads_started":1}
I20260807 18:28:21.404636 1056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab: Bootstrap starting.
I20260807 18:28:21.407114 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:28:21.407977 1056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:21.408931 1056 log.cc:824] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:21.411677 1056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab: No bootstrap required, opened a new log
I20260807 18:28:21.418659 1056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f5a75a628745ea9aab5086756d60ab" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34197 } }
I20260807 18:28:21.419062 1056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:21.419212 1056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f5a75a628745ea9aab5086756d60ab, State: Initialized, Role: FOLLOWER
I20260807 18:28:21.419716 1056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f5a75a628745ea9aab5086756d60ab" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34197 } }
I20260807 18:28:21.419960 1056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:21.420110 1056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:21.420291 1056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:21.422416 1056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f5a75a628745ea9aab5086756d60ab" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34197 } }
I20260807 18:28:21.422925 1056 leader_election.cc:302] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80f5a75a628745ea9aab5086756d60ab; no voters:
I20260807 18:28:21.423576 1056 leader_election.cc:288] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:21.423703 1061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:21.424427 1061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [term 1 LEADER]: Becoming Leader. State: Replica: 80f5a75a628745ea9aab5086756d60ab, State: Running, Role: LEADER
I20260807 18:28:21.424904 1061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f5a75a628745ea9aab5086756d60ab" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34197 } }
I20260807 18:28:21.425659 1056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:21.427847 1064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80f5a75a628745ea9aab5086756d60ab. Latest consensus state: current_term: 1 leader_uuid: "80f5a75a628745ea9aab5086756d60ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f5a75a628745ea9aab5086756d60ab" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34197 } } }
I20260807 18:28:21.428192 1064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:21.428643 1062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80f5a75a628745ea9aab5086756d60ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f5a75a628745ea9aab5086756d60ab" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34197 } } }
I20260807 18:28:21.428920 1062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:21.430220 1069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:21.436560 1069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:21.445727 1069 catalog_manager.cc:1347] Generated new cluster ID: 90efd16883c247f2a896265ef77ac64b
I20260807 18:28:21.445871 1069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:21.462158 1069 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:21.463995 1069 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:21.473953 1069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80f5a75a628745ea9aab5086756d60ab: Generated new TSK 0
I20260807 18:28:21.475001 1069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:28:22.375005 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:21.497754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.27.1.254:34197)
0807 18:28:21.497994 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:21.498002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:21.498400 (+ 398us) server_negotiation.cc:407] Connection header received
0807 18:28:21.498617 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:21.498620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:21.498688 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:21.498808 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0807 18:28:21.499343 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:21.500087 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:21.502111 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:21.502572 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:21.502669 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:21.503292 (+ 623us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 18:28:22.374414 (+871122us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 18:28:22.374751 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58314: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 18:28:22.375195 1059 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58314: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260807 18:28:22.375582 1091 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:21.497658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34197 (local address 127.0.0.1:58314)
0807 18:28:21.498268 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:21.498289 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:21.498443 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:21.498847 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:21.498853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:21.498866 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0807 18:28:21.499192 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:21.499207 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:21.500227 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:21.500235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:21.501975 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:21.501991 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:21.502737 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:21.502744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:21.502944 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:21.502989 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 18:28:22.374708 (+871719us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 18:28:22.374947 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.1.254:34197: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":331,"threads_started":1}
I20260807 18:28:22.377127 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 983
I20260807 18:28:22.378255 1050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:22.480397 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 983
2026-08-07T18:28:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1625 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (425 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (425 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:28:23 dist-test-slave-1xrh krb5kdc[1107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:28:23 dist-test-slave-1xrh krb5kdc[1107](info): set up 2 sockets
Aug 07 18:28:23 dist-test-slave-1xrh krb5kdc[1107](info): commencing operation
krb5kdc: starting...
W20260807 18:28:24.478118 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:28:24 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:28:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:28:24.710796 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:42621
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:42621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:25.036860 1122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:25.037192 1122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:25.037360 1122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:25.044471 1122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:25.044616 1122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:25.044721 1122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:25.044821 1122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:25.044902 1122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:25.054688 1122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:42621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:42621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:25.056120 1122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:25.057430 1122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:25.069190 1129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:25.069223 1130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:25.069149 1132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:25.069937 1122 server_base.cc:1034] running on GCE node
I20260807 18:28:25.070763 1122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:25.072247 1122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:25.073467 1122 hybrid_clock.cc:648] HybridClock initialized: now 1786127305073429 us; error 51 us; skew 500 ppm
I20260807 18:28:25.073802 1122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:25 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127305, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:25.076921 1122 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:28:25.078323 1122 webserver.cc:460] Webserver started at http://127.27.1.254:34807/ using document root <none> and password file <none>
I20260807 18:28:25.079017 1122 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:25.079187 1122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:25.079504 1122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:25.082432 1122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "c996cb984e6f4b73bccf0aa7c7507e19"
format_stamp: "Formatted at 2026-08-07 18:28:25 on dist-test-slave-1xrh"
server_key: "bb973d45985aff20d4fd0886059bdc4e"
server_key_iv: "7674b3cc04e9e35f1ad2ed04b7eb3647"
server_key_version: "encryptionkey@0"
I20260807 18:28:25.083284 1122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "c996cb984e6f4b73bccf0aa7c7507e19"
format_stamp: "Formatted at 2026-08-07 18:28:25 on dist-test-slave-1xrh"
server_key: "bb973d45985aff20d4fd0886059bdc4e"
server_key_iv: "7674b3cc04e9e35f1ad2ed04b7eb3647"
server_key_version: "encryptionkey@0"
I20260807 18:28:25.088479 1122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:28:25.091825 1139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:25.093081 1122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:28:25.093259 1122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "c996cb984e6f4b73bccf0aa7c7507e19"
format_stamp: "Formatted at 2026-08-07 18:28:25 on dist-test-slave-1xrh"
server_key: "bb973d45985aff20d4fd0886059bdc4e"
server_key_iv: "7674b3cc04e9e35f1ad2ed04b7eb3647"
server_key_version: "encryptionkey@0"
I20260807 18:28:25.093459 1122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:25.111505 1122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:25.115497 1122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:25.115772 1122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:25.134075 1122 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:42621
I20260807 18:28:25.134099 1194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:42621 every 8 connection(s)
I20260807 18:28:25.135432 1122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:25.138878 1195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:25.142537 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 1122
I20260807 18:28:25.142750 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:25.143641 27655 external_mini_cluster.cc:1428] Setting key 91bd176fb270d50afed722ac2fb1f664
I20260807 18:28:25.151477 1195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19: Bootstrap starting.
I20260807 18:28:25.154315 1195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:25.155332 1195 log.cc:824] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19: Log is configured to *not* fsync() on all Append() calls
Aug 07 18:28:25 dist-test-slave-1xrh krb5kdc[1107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:25.157944 1195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19: No bootstrap required, opened a new log
I20260807 18:28:25.165097 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:25.145650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.27.1.254:42621)
0807 18:28:25.146404 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:25.146415 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:25.146455 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:28:25.148217 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:25.148239 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:25.148616 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:25.149080 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:25.150544 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:25.151936 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:25.153148 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:25.153773 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:25.153974 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:25.157860 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:25.157875 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:25.157889 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:25.157923 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:25.161111 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:25.161813 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:25.161829 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:25.161834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:25.162012 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:25.162423 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:25.162428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:25.162431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:25.162944 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:25.163289 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:25.163712 (+ 423us) server_negotiation.cc:299] Negotiation successful
0807 18:28:25.164151 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":343,"threads_started":1}
I20260807 18:28:25.165740 1195 raft_consensus.cc:348] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c996cb984e6f4b73bccf0aa7c7507e19" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42621 } }
I20260807 18:28:25.166157 1195 raft_consensus.cc:374] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:25.166272 1195 raft_consensus.cc:729] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c996cb984e6f4b73bccf0aa7c7507e19, State: Initialized, Role: FOLLOWER
I20260807 18:28:25.166860 1195 consensus_queue.cc:260] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c996cb984e6f4b73bccf0aa7c7507e19" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42621 } }
I20260807 18:28:25.167166 1195 raft_consensus.cc:388] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:25.167325 1195 raft_consensus.cc:482] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:25.167510 1195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:25.169714 1195 raft_consensus.cc:504] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c996cb984e6f4b73bccf0aa7c7507e19" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42621 } }
I20260807 18:28:25.170187 1195 leader_election.cc:302] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c996cb984e6f4b73bccf0aa7c7507e19; no voters:
I20260807 18:28:25.170753 1195 leader_election.cc:288] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:25.170986 1201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:25.172012 1201 raft_consensus.cc:686] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [term 1 LEADER]: Becoming Leader. State: Replica: c996cb984e6f4b73bccf0aa7c7507e19, State: Running, Role: LEADER
I20260807 18:28:25.172636 1201 consensus_queue.cc:237] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c996cb984e6f4b73bccf0aa7c7507e19" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42621 } }
I20260807 18:28:25.172880 1195 sys_catalog.cc:564] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:25.175621 1203 sys_catalog.cc:455] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c996cb984e6f4b73bccf0aa7c7507e19. Latest consensus state: current_term: 1 leader_uuid: "c996cb984e6f4b73bccf0aa7c7507e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c996cb984e6f4b73bccf0aa7c7507e19" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42621 } } }
I20260807 18:28:25.175771 1202 sys_catalog.cc:455] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c996cb984e6f4b73bccf0aa7c7507e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c996cb984e6f4b73bccf0aa7c7507e19" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42621 } } }
I20260807 18:28:25.176164 1202 sys_catalog.cc:458] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:25.176054 1203 sys_catalog.cc:458] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:25.183017 1212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:25.188552 1212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:25.197180 1212 catalog_manager.cc:1347] Generated new cluster ID: 563b1a1fff4542b5bb38e1ffcee9ffcd
I20260807 18:28:25.197316 1212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:25.217639 1212 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:25.218924 1212 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:25.231964 1212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c996cb984e6f4b73bccf0aa7c7507e19: Generated new TSK 0
I20260807 18:28:25.232754 1212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:28:25.294620 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42621
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:25.655036 1225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:25.655414 1225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:25.655586 1225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:25.662799 1225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:25.662949 1225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:25.663157 1225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:25.673152 1225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:25.674582 1225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:25.676141 1225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:25.689363 1233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:25.689400 1231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:25.689589 1230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:25.690379 1225 server_base.cc:1034] running on GCE node
I20260807 18:28:25.691022 1225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:25.691977 1225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:25.693181 1225 hybrid_clock.cc:648] HybridClock initialized: now 1786127305693147 us; error 32 us; skew 500 ppm
I20260807 18:28:25.693492 1225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:25 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127305, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:25.696642 1225 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:28:25.698115 1225 webserver.cc:460] Webserver started at http://127.27.1.193:43897/ using document root <none> and password file <none>
I20260807 18:28:25.698853 1225 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:25.699002 1225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:25.699363 1225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:25.702193 1225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "9b9dbb8838094f4e9c328f0cdcf41f60"
format_stamp: "Formatted at 2026-08-07 18:28:25 on dist-test-slave-1xrh"
server_key: "15e7e84a2c9dec7838089b5a29e92962"
server_key_iv: "373bf0d9d6f012887dacddf3b8dbbd85"
server_key_version: "encryptionkey@0"
I20260807 18:28:25.702948 1225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "9b9dbb8838094f4e9c328f0cdcf41f60"
format_stamp: "Formatted at 2026-08-07 18:28:25 on dist-test-slave-1xrh"
server_key: "15e7e84a2c9dec7838089b5a29e92962"
server_key_iv: "373bf0d9d6f012887dacddf3b8dbbd85"
server_key_version: "encryptionkey@0"
I20260807 18:28:25.707995 1225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:28:25.711270 1241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:25.712530 1225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:25.712706 1225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "9b9dbb8838094f4e9c328f0cdcf41f60"
format_stamp: "Formatted at 2026-08-07 18:28:25 on dist-test-slave-1xrh"
server_key: "15e7e84a2c9dec7838089b5a29e92962"
server_key_iv: "373bf0d9d6f012887dacddf3b8dbbd85"
server_key_version: "encryptionkey@0"
I20260807 18:28:25.712899 1225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:25.737011 1225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:25.740896 1225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:25.741206 1225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:25.742218 1225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:25.743592 1225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:25.743693 1225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:25.743847 1225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:25.743920 1225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:25.777136 1225 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:32853
I20260807 18:28:25.777180 1361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:32853 every 8 connection(s)
I20260807 18:28:25.778491 1225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:25.780413 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 1225
I20260807 18:28:25.780615 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:28:25.781430 27655 external_mini_cluster.cc:1428] Setting key 3fcdc26006b7c6521222b17003c30348
Aug 07 18:28:25 dist-test-slave-1xrh krb5kdc[1107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127305, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:25.802958 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:25.781203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:51351 (local address 127.27.1.254:42621)
0807 18:28:25.781680 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:25.781688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:25.782914 (+ 1226us) server_negotiation.cc:407] Connection header received
0807 18:28:25.785658 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:25.785662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:25.785725 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:25.785823 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:25.788050 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:25.788753 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:25.791255 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:25.791650 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:25.791741 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:25.795809 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:25.795819 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:25.795826 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:25.795872 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:25.799503 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:25.800250 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:25.800259 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:25.800266 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:25.800419 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:25.800828 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:25.800836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:25.800842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:25.801341 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:25.801528 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:25.802187 (+ 659us) server_negotiation.cc:299] Negotiation successful
0807 18:28:25.802441 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260807 18:28:25.803498 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:25.781786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42621 (local address 127.27.1.193:51351)
0807 18:28:25.782673 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:25.782773 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:25.785333 (+ 2560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:25.786102 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:25.786117 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:25.787011 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:25.787858 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:25.787887 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:25.788913 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:25.788921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:25.791084 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:25.791102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:25.791869 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:25.791878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:25.792196 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:25.792962 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:25.792996 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:25.795571 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:25.799628 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:25.799634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:25.799646 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:25.800107 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:25.800530 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:25.800534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:25.800536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:25.800709 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:25.801480 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:25.801486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:25.801763 (+ 277us) client_negotiation.cc:769] Sending connection context
0807 18:28:25.801992 (+ 229us) client_negotiation.cc:240] Negotiation successful
0807 18:28:25.802351 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":394,"threads_started":1}
I20260807 18:28:25.805126 1362 heartbeater.cc:347] Connected to a master server at 127.27.1.254:42621
I20260807 18:28:25.805531 1362 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:25.806514 1362 heartbeater.cc:511] Master 127.27.1.254:42621 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
I20260807 18:28:25.808836 1157 ts_manager.cc:194] Registered new tserver with Master: 9b9dbb8838094f4e9c328f0cdcf41f60 (127.27.1.193:32853)
I20260807 18:28:25.810322 1157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:51351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:28:25.840237 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42621
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:26.175514 1368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:26.175782 1368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:26.175938 1368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:26.183380 1368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:26.183499 1368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:26.183696 1368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:26.193596 1368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:26.194908 1368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:26.196256 1368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:26.211547 1373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:26.211489 1376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:26.211508 1374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:26.212365 1368 server_base.cc:1034] running on GCE node
I20260807 18:28:26.212903 1368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:26.213733 1368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:26.214938 1368 hybrid_clock.cc:648] HybridClock initialized: now 1786127306214902 us; error 34 us; skew 500 ppm
I20260807 18:28:26.215299 1368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:26.218564 1368 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:28:26.219998 1368 webserver.cc:460] Webserver started at http://127.27.1.194:39873/ using document root <none> and password file <none>
I20260807 18:28:26.220680 1368 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:26.220860 1368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:26.221172 1368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:26.224104 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "063fafef175042bbae22abc7fb91d705"
format_stamp: "Formatted at 2026-08-07 18:28:26 on dist-test-slave-1xrh"
server_key: "6a5e96ee631606b4961d2ac54caa1cc5"
server_key_iv: "b29b68925fc6fe66a2a6129015ce6c6b"
server_key_version: "encryptionkey@0"
I20260807 18:28:26.224880 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "063fafef175042bbae22abc7fb91d705"
format_stamp: "Formatted at 2026-08-07 18:28:26 on dist-test-slave-1xrh"
server_key: "6a5e96ee631606b4961d2ac54caa1cc5"
server_key_iv: "b29b68925fc6fe66a2a6129015ce6c6b"
server_key_version: "encryptionkey@0"
I20260807 18:28:26.229861 1368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 18:28:26.233227 1383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:26.234537 1368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:28:26.234714 1368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "063fafef175042bbae22abc7fb91d705"
format_stamp: "Formatted at 2026-08-07 18:28:26 on dist-test-slave-1xrh"
server_key: "6a5e96ee631606b4961d2ac54caa1cc5"
server_key_iv: "b29b68925fc6fe66a2a6129015ce6c6b"
server_key_version: "encryptionkey@0"
I20260807 18:28:26.234917 1368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:26.263553 1368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:26.267724 1368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:26.268019 1368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:26.268981 1368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:26.270282 1368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:26.270391 1368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:26.270569 1368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:26.270645 1368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:26.304700 1368 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:41779
I20260807 18:28:26.304745 1505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:41779 every 8 connection(s)
I20260807 18:28:26.305969 1368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:26.313791 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 1368
I20260807 18:28:26.313964 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:28:26.314424 27655 external_mini_cluster.cc:1428] Setting key 4074bcc4493c2c9ebc3700ef668036ef
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:26.327728 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:26.308493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:39713 (local address 127.27.1.254:42621)
0807 18:28:26.308922 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:26.308926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:26.309892 (+ 966us) server_negotiation.cc:407] Connection header received
0807 18:28:26.311791 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:26.311798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:26.311890 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:26.312019 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:26.313556 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.314420 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.316470 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.317049 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.317224 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:26.321603 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:26.321608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:26.321612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:26.321645 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:26.324463 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.325231 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.325236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.325238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.325361 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.325800 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.325812 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.325814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.326076 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:26.326257 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:26.327067 (+ 810us) server_negotiation.cc:299] Negotiation successful
0807 18:28:26.327343 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":231,"threads_started":1}
I20260807 18:28:26.328711 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:26.308926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42621 (local address 127.27.1.194:39713)
0807 18:28:26.309738 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:26.309782 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:26.311525 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:26.312240 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:26.312248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:26.312752 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:26.313349 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:26.313367 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.314621 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.314626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:26.316214 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:26.316231 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.317314 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.317321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:26.317668 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:26.318465 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:26.318508 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:26.321335 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:26.324621 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:26.324630 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:26.324648 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:26.325092 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:26.325484 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:26.325489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:26.325492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:26.325678 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:26.326229 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:26.326239 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:26.326607 (+ 368us) client_negotiation.cc:769] Sending connection context
0807 18:28:26.326963 (+ 356us) client_negotiation.cc:240] Negotiation successful
0807 18:28:26.327539 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":296,"threads_started":1}
I20260807 18:28:26.330132 1506 heartbeater.cc:347] Connected to a master server at 127.27.1.254:42621
I20260807 18:28:26.330470 1506 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:26.331446 1506 heartbeater.cc:511] Master 127.27.1.254:42621 requested a full tablet report, sending...
I20260807 18:28:26.333012 1157 ts_manager.cc:194] Registered new tserver with Master: 063fafef175042bbae22abc7fb91d705 (127.27.1.194:41779)
I20260807 18:28:26.333817 1157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:39713
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:28:26.374711 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42621
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:26.702381 1512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:26.702682 1512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:26.702854 1512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:26.709743 1512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:26.709882 1512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:26.710062 1512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:26.719442 1512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:26.720669 1512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:26.721952 1512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:26.735037 1517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:26.735081 1518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:26.735165 1520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:26.735787 1512 server_base.cc:1034] running on GCE node
I20260807 18:28:26.736342 1512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:26.737248 1512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:26.738468 1512 hybrid_clock.cc:648] HybridClock initialized: now 1786127306738413 us; error 49 us; skew 500 ppm
I20260807 18:28:26.738791 1512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:26.742079 1512 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:28:26.743652 1512 webserver.cc:460] Webserver started at http://127.27.1.195:46479/ using document root <none> and password file <none>
I20260807 18:28:26.744415 1512 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:26.744556 1512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:26.744884 1512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:26.747808 1512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "fd79dafed36c4dc198449b84ccf8ffa7"
format_stamp: "Formatted at 2026-08-07 18:28:26 on dist-test-slave-1xrh"
server_key: "85fdeaa27ff58615063fcd300299f51f"
server_key_iv: "84c53366caf6023d7c957ba932e5cbc8"
server_key_version: "encryptionkey@0"
I20260807 18:28:26.748605 1512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "fd79dafed36c4dc198449b84ccf8ffa7"
format_stamp: "Formatted at 2026-08-07 18:28:26 on dist-test-slave-1xrh"
server_key: "85fdeaa27ff58615063fcd300299f51f"
server_key_iv: "84c53366caf6023d7c957ba932e5cbc8"
server_key_version: "encryptionkey@0"
I20260807 18:28:26.753897 1512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:28:26.757213 1527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:26.758502 1512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:28:26.758679 1512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "fd79dafed36c4dc198449b84ccf8ffa7"
format_stamp: "Formatted at 2026-08-07 18:28:26 on dist-test-slave-1xrh"
server_key: "85fdeaa27ff58615063fcd300299f51f"
server_key_iv: "84c53366caf6023d7c957ba932e5cbc8"
server_key_version: "encryptionkey@0"
I20260807 18:28:26.758878 1512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:26.788169 1512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:26.792600 1512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:26.792917 1512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:26.793908 1512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:26.795351 1512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:26.795461 1512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:26.795612 1512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:26.795686 1512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:26.813390 1362 heartbeater.cc:503] Master 127.27.1.254:42621 was elected leader, sending a full tablet report...
I20260807 18:28:26.829211 1512 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:33265
I20260807 18:28:26.829257 1652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:33265 every 8 connection(s)
I20260807 18:28:26.830480 1512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:26.838032 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 1512
I20260807 18:28:26.838212 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:28:26.838685 27655 external_mini_cluster.cc:1428] Setting key afd7c08855dfac3f2c15e71a28b3df35
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:26.851032 1655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:26.833223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:58705 (local address 127.27.1.254:42621)
0807 18:28:26.833702 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:26.833716 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:26.834597 (+ 881us) server_negotiation.cc:407] Connection header received
0807 18:28:26.836434 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:26.836438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:26.836518 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:26.836615 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:26.838092 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.839058 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.840786 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.841246 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.841403 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:26.844755 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:26.844760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:26.844764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:26.844795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:26.847421 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.848245 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.848253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.848256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.848424 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.848850 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.848859 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.848862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.849206 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:26.849375 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:26.850413 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0807 18:28:26.850652 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260807 18:28:26.851861 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:26.833698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42621 (local address 127.27.1.195:58705)
0807 18:28:26.834430 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:26.834475 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:26.836193 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:26.836802 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:26.836810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:26.837301 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:26.837856 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:26.837881 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.839221 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.839228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:26.840625 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:26.840641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.841460 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.841469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:26.841714 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:26.842361 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:26.842382 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:26.844539 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:26.847568 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:26.847578 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:26.847595 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:26.848130 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:26.848548 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:26.848553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:26.848559 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:26.848735 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:26.849450 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:26.849456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:26.849913 (+ 457us) client_negotiation.cc:769] Sending connection context
0807 18:28:26.850281 (+ 368us) client_negotiation.cc:240] Negotiation successful
0807 18:28:26.850660 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":292,"threads_started":1}
I20260807 18:28:26.853305 1653 heartbeater.cc:347] Connected to a master server at 127.27.1.254:42621
I20260807 18:28:26.853645 1653 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:26.854395 1653 heartbeater.cc:511] Master 127.27.1.254:42621 requested a full tablet report, sending...
I20260807 18:28:26.855832 1157 ts_manager.cc:194] Registered new tserver with Master: fd79dafed36c4dc198449b84ccf8ffa7 (127.27.1.195:33265)
I20260807 18:28:26.856619 1157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:58705
I20260807 18:28:26.865919 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/test-admin.keytab.
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:26.896880 27655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 18:28:26.902174 27655 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5cc' of type FILE
W20260807 18:28:26.902231 1662 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:26.899592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.27.1.193:32853)
0807 18:28:26.900234 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:26.900241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:26.900269 (+ 28us) server_negotiation.cc:407] Connection header received
0807 18:28:26.900342 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:26.900349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:26.900553 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:26.900712 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:26.901775 (+ 1063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38586: BlockingRecv error: recv got EOF from 127.0.0.1:38586 (error 108)
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":316,"threads_started":1}
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:26.903985 27655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:26.905297 27655 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:28:26.920228 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:26.906831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.27.1.193:32853)
0807 18:28:26.907052 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:26.907058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:26.907314 (+ 256us) server_negotiation.cc:407] Connection header received
0807 18:28:26.907433 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:26.907436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:26.907491 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:26.907569 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:26.908526 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.909517 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.910674 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.911288 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.911391 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:26.914815 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:26.914826 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:26.914859 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:26.914903 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:26.917830 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.918437 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.918443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.918445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.918576 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.918881 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.918884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.918886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.919288 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:26.919511 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:26.919777 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 18:28:26.919961 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260807 18:28:26.926041 1662 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:26.924394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.27.1.193:32853)
0807 18:28:26.924594 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:26.924600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:26.924776 (+ 176us) server_negotiation.cc:407] Connection header received
0807 18:28:26.924993 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:26.924998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:26.925056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:26.925151 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:26.925811 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38604: BlockingRecv error: recv got EOF from 127.0.0.1:38604 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 18:28:26.926323 27655 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5cc' of type FILE
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:26.927767 27655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:26.929030 27655 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:28:26.942853 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:26.930229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.27.1.193:32853)
0807 18:28:26.930384 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:26.930388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:26.930760 (+ 372us) server_negotiation.cc:407] Connection header received
0807 18:28:26.930882 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:26.930885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:26.930931 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:26.931005 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:26.932062 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.932725 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.933903 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.934446 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.934556 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:26.938069 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:26.938073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:26.938077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:26.938171 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:26.940752 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.941303 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.941306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.941308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.941416 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.941733 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.941736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.941738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.941971 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:26.942160 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:26.942391 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 18:28:26.942584 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260807 18:28:26.946815 1662 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:26.945259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.27.1.193:32853)
0807 18:28:26.945398 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:26.945401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:26.945741 (+ 340us) server_negotiation.cc:407] Connection header received
0807 18:28:26.945880 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:26.945883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:26.945942 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:26.946016 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:26.946581 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38608: BlockingRecv error: recv got EOF from 127.0.0.1:38608 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260807 18:28:26.947046 27655 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786127255008662-27655-0/krb5kdc/krb5cc' of type FILE
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:26.948496 27655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:26.949854 27655 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 18:28:26 dist-test-slave-1xrh krb5kdc[1107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:28:26.965118 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:26.951088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.27.1.193:32853)
0807 18:28:26.951282 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:26.951288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:26.951777 (+ 489us) server_negotiation.cc:407] Connection header received
0807 18:28:26.951921 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:26.951924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:26.951971 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:26.952050 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:26.953048 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.953713 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.954976 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.955463 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.955555 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:26.959285 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:26.959290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:26.959293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:26.959320 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:26.962538 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.963210 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.963216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.963220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.963374 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.963724 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.963728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.963730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.964089 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:26.964346 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:26.964524 (+ 178us) server_negotiation.cc:299] Negotiation successful
0807 18:28:26.964741 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 18:28:26.966691 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 1225
I20260807 18:28:26.978363 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:26.967420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.27.1.193:32853)
0807 18:28:26.967629 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:26.967637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:26.967844 (+ 207us) server_negotiation.cc:407] Connection header received
0807 18:28:26.968062 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:26.968066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:26.968113 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:26.968184 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:26.969149 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.969770 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.971035 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:26.971465 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:26.971556 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:26.973819 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:26.973823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:26.973826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:26.973852 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:26.976166 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.976800 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.976804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.976806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.976914 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:26.977227 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:26.977230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:26.977232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:26.977440 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:26.977618 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:26.977853 (+ 235us) server_negotiation.cc:299] Negotiation successful
0807 18:28:26.978020 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 18:28:26.979044 1357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:27.065248 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 1225
I20260807 18:28:27.084414 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 1368
Aug 07 18:28:27 dist-test-slave-1xrh krb5kdc[1107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:28:27.099696 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:27.085342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.27.1.194:41779)
0807 18:28:27.085900 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:27.085907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:27.085931 (+ 24us) server_negotiation.cc:407] Connection header received
0807 18:28:27.086000 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:27.086006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:27.086208 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:27.086329 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:27.087344 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:27.088333 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:27.089544 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:27.090072 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:27.090170 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:27.094092 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:27.094103 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:27.094116 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:27.094148 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:27.097049 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:27.097684 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:27.097690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:27.097693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:27.097807 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:27.098121 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:27.098124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:27.098126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:27.098570 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:27.098810 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:27.099112 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 18:28:27.099383 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":261,"threads_started":1}
I20260807 18:28:27.101011 1501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:27.183252 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 1368
I20260807 18:28:27.202695 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 1512
Aug 07 18:28:27 dist-test-slave-1xrh krb5kdc[1107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:28:27.217756 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:27.203754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.27.1.195:33265)
0807 18:28:27.204412 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:27.204419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:27.204448 (+ 29us) server_negotiation.cc:407] Connection header received
0807 18:28:27.204510 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:27.204517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:27.204723 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:27.204845 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:27.205833 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:27.206846 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:27.208093 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:27.208639 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:27.208734 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:27.212565 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:27.212576 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:27.212590 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:27.212625 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:27.215441 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:27.216082 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:27.216089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:27.216091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:27.216205 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:27.216522 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:27.216525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:27.216527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:27.216836 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:27.217073 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:27.217274 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 18:28:27.217462 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":340,"threads_started":1}
I20260807 18:28:27.219060 1648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:27.301972 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 1512
I20260807 18:28:27.325613 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 1122
I20260807 18:28:27.327105 1190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:27.431835 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 1122
2026-08-07T18:28:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4107 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:28:27 dist-test-slave-1xrh krb5kdc[1683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:28:27 dist-test-slave-1xrh krb5kdc[1683](info): set up 2 sockets
Aug 07 18:28:27 dist-test-slave-1xrh krb5kdc[1683](info): commencing operation
krb5kdc: starting...
W20260807 18:28:28.574458 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:28:28 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:28:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.254.keytab.
Entry for principal oryx/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.254.keytab.
I20260807 18:28:28.810513 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38033
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.254.keytab
--principal=oryx/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:38033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:29.132153 1697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:29.132436 1697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:29.132611 1697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:29.139731 1697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:29.139829 1697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:29.139927 1697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:29.140025 1697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:29.140105 1697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:29.149878 1697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:38033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.254.keytab
--principal=oryx/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:29.151283 1697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:29.152582 1697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:29.164340 1703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:29.164433 1704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:29.164479 1706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:29.165696 1697 server_base.cc:1034] running on GCE node
I20260807 18:28:29.166462 1697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:29.167918 1697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:29.169150 1697 hybrid_clock.cc:648] HybridClock initialized: now 1786127309169112 us; error 50 us; skew 500 ppm
I20260807 18:28:29.169488 1697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:29 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127309, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:29.176574 1697 init.cc:377] Logged in from keytab as oryx/127.27.1.254@KRBTEST.COM (short username oryx)
I20260807 18:28:29.178149 1697 webserver.cc:460] Webserver started at http://127.27.1.254:44049/ using document root <none> and password file <none>
I20260807 18:28:29.178846 1697 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:29.178993 1697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:29.179366 1697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:29.182377 1697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "c98333801763413d99a0d26660b81d1a"
format_stamp: "Formatted at 2026-08-07 18:28:29 on dist-test-slave-1xrh"
server_key: "126d5e4516ede39b59f5b9768dbe1a86"
server_key_iv: "a782a4c8521689d9b728dafe1f04c1ce"
server_key_version: "encryptionkey@0"
I20260807 18:28:29.183221 1697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "c98333801763413d99a0d26660b81d1a"
format_stamp: "Formatted at 2026-08-07 18:28:29 on dist-test-slave-1xrh"
server_key: "126d5e4516ede39b59f5b9768dbe1a86"
server_key_iv: "a782a4c8521689d9b728dafe1f04c1ce"
server_key_version: "encryptionkey@0"
I20260807 18:28:29.188222 1697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 18:28:29.191514 1714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:29.192760 1697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 18:28:29.192950 1697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "c98333801763413d99a0d26660b81d1a"
format_stamp: "Formatted at 2026-08-07 18:28:29 on dist-test-slave-1xrh"
server_key: "126d5e4516ede39b59f5b9768dbe1a86"
server_key_iv: "a782a4c8521689d9b728dafe1f04c1ce"
server_key_version: "encryptionkey@0"
I20260807 18:28:29.193156 1697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:29.223385 1697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:29.227857 1697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:29.228155 1697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:29.246846 1697 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:38033
I20260807 18:28:29.246884 1771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:38033 every 8 connection(s)
I20260807 18:28:29.248445 1697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:29.251314 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 1697
I20260807 18:28:29.251535 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:29.252358 27655 external_mini_cluster.cc:1428] Setting key 3847746f3cc7c9b173df935ca79430ac
I20260807 18:28:29.252969 1773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:29.265033 1773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a: Bootstrap starting.
Aug 07 18:28:29 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:29.268615 1773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:29.269695 1773 log.cc:824] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:29.273020 1773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a: No bootstrap required, opened a new log
I20260807 18:28:29.275425 1776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:29.254506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.27.1.254:38033)
0807 18:28:29.255375 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:29.255386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:29.255427 (+ 41us) server_negotiation.cc:407] Connection header received
0807 18:28:29.257615 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:29.257636 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:29.258022 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:29.258424 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:29.259942 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:29.261159 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:29.262361 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:29.263063 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:29.263304 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:29.267678 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:29.267686 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:29.267699 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:29.267739 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:29.271367 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:29.272136 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:29.272142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:29.272148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:29.272298 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:29.272719 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:29.272722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:29.272725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:29.273277 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:29.273499 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:29.273999 (+ 500us) server_negotiation.cc:299] Negotiation successful
0807 18:28:29.274324 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":387,"threads_started":1}
I20260807 18:28:29.280877 1773 raft_consensus.cc:348] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } }
I20260807 18:28:29.281226 1773 raft_consensus.cc:374] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:29.281339 1773 raft_consensus.cc:729] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c98333801763413d99a0d26660b81d1a, State: Initialized, Role: FOLLOWER
I20260807 18:28:29.281884 1773 consensus_queue.cc:260] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } }
I20260807 18:28:29.282124 1773 raft_consensus.cc:388] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:29.282275 1773 raft_consensus.cc:482] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:29.282466 1773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:29.284866 1773 raft_consensus.cc:504] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } }
I20260807 18:28:29.285338 1773 leader_election.cc:302] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c98333801763413d99a0d26660b81d1a; no voters:
I20260807 18:28:29.285910 1773 leader_election.cc:288] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:29.286036 1778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:29.286777 1778 raft_consensus.cc:686] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 1 LEADER]: Becoming Leader. State: Replica: c98333801763413d99a0d26660b81d1a, State: Running, Role: LEADER
I20260807 18:28:29.287398 1778 consensus_queue.cc:237] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } }
I20260807 18:28:29.288097 1773 sys_catalog.cc:564] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:29.290576 1779 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } }
I20260807 18:28:29.290550 1781 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c98333801763413d99a0d26660b81d1a. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } }
I20260807 18:28:29.291021 1779 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:29.291021 1781 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:29.298393 1788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:29.303454 1788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:29.314761 1788 catalog_manager.cc:1347] Generated new cluster ID: 1bcd0fbcc44d441687caedeef987db55
I20260807 18:28:29.314960 1788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:29.332044 1788 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:29.333376 1788 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:29.344902 1788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a: Generated new TSK 0
I20260807 18:28:29.345760 1788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab.
Entry for principal oryx/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab.
I20260807 18:28:29.412859 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38033
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab
--principal=oryx/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:29.749884 1802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:29.750165 1802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:29.750339 1802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:29.757793 1802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:29.757901 1802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:29.758078 1802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:29.767758 1802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab
--principal=oryx/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:29.769140 1802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:29.770521 1802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:29.784040 1807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:29.784201 1808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:29.784171 1810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:29.784816 1802 server_base.cc:1034] running on GCE node
I20260807 18:28:29.785383 1802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:29.786321 1802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:29.787521 1802 hybrid_clock.cc:648] HybridClock initialized: now 1786127309787482 us; error 31 us; skew 500 ppm
I20260807 18:28:29.787849 1802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:29 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127309, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:29.791224 1802 init.cc:377] Logged in from keytab as oryx/127.27.1.193@KRBTEST.COM (short username oryx)
I20260807 18:28:29.792827 1802 webserver.cc:460] Webserver started at http://127.27.1.193:44231/ using document root <none> and password file <none>
I20260807 18:28:29.793535 1802 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:29.793684 1802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:29.794008 1802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:29.797418 1802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "ebf04659d5b944b2ab47b4a7d0f8e918"
format_stamp: "Formatted at 2026-08-07 18:28:29 on dist-test-slave-1xrh"
server_key: "fa4ae1039df1c1b875648e586a289b35"
server_key_iv: "78bf4212ec73ab8903687137b3f6f2cd"
server_key_version: "encryptionkey@0"
I20260807 18:28:29.798297 1802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "ebf04659d5b944b2ab47b4a7d0f8e918"
format_stamp: "Formatted at 2026-08-07 18:28:29 on dist-test-slave-1xrh"
server_key: "fa4ae1039df1c1b875648e586a289b35"
server_key_iv: "78bf4212ec73ab8903687137b3f6f2cd"
server_key_version: "encryptionkey@0"
I20260807 18:28:29.803505 1802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 18:28:29.807101 1818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:29.808445 1802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:28:29.808630 1802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "ebf04659d5b944b2ab47b4a7d0f8e918"
format_stamp: "Formatted at 2026-08-07 18:28:29 on dist-test-slave-1xrh"
server_key: "fa4ae1039df1c1b875648e586a289b35"
server_key_iv: "78bf4212ec73ab8903687137b3f6f2cd"
server_key_version: "encryptionkey@0"
I20260807 18:28:29.808836 1802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:29.833546 1802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:29.837700 1802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:29.838044 1802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:29.839023 1802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:29.840330 1802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:29.840433 1802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:29.840590 1802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:29.840673 1802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:29.875248 1802 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:35689
I20260807 18:28:29.875303 1937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:35689 every 8 connection(s)
I20260807 18:28:29.876555 1802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:29.886899 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 1802
I20260807 18:28:29.887111 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:28:29.887976 27655 external_mini_cluster.cc:1428] Setting key d060cb29b7dbeb925f4ea4724002b11f
Aug 07 18:28:29 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127309, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:29.897200 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:29.879106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:44095 (local address 127.27.1.254:38033)
0807 18:28:29.879635 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:29.879647 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:29.880449 (+ 802us) server_negotiation.cc:407] Connection header received
0807 18:28:29.882261 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:29.882267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:29.882359 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:29.882485 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:29.884030 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:29.884736 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:29.886263 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:29.886658 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:29.886790 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:29.891183 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:29.891188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:29.891192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:29.891223 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:29.893757 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:29.894724 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:29.894728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:29.894730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:29.894845 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:29.895370 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:29.895375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:29.895377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:29.895691 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:29.895901 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:29.896597 (+ 696us) server_negotiation.cc:299] Negotiation successful
0807 18:28:29.896815 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":228,"threads_started":1}
I20260807 18:28:29.898221 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:29.879596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.193:44095)
0807 18:28:29.880282 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:29.880320 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:29.881998 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:29.882710 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:29.882719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:29.883278 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:29.883852 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:29.883870 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:29.884892 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:29.884900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:29.886111 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:29.886123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:29.886898 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:29.886905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:29.887254 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:29.888060 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:29.888081 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:29.890932 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:29.893899 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:29.893910 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:29.893928 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:29.894594 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:29.894980 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:29.894986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:29.894998 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:29.895258 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:29.895849 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:29.895857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:29.896199 (+ 342us) client_negotiation.cc:769] Sending connection context
0807 18:28:29.896478 (+ 279us) client_negotiation.cc:240] Negotiation successful
0807 18:28:29.897086 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":268,"threads_started":1}
I20260807 18:28:29.899834 1938 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38033
I20260807 18:28:29.900151 1938 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:29.900856 1938 heartbeater.cc:511] Master 127.27.1.254:38033 requested a full tablet report, sending...
I20260807 18:28:29.902853 1732 ts_manager.cc:194] Registered new tserver with Master: ebf04659d5b944b2ab47b4a7d0f8e918 (127.27.1.193:35689)
I20260807 18:28:29.904323 1732 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:44095
WARNING: no policy specified for oryx/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab.
Entry for principal oryx/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab.
I20260807 18:28:29.942320 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38033
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab
--principal=oryx/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:30.269496 1944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:30.269760 1944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:30.269932 1944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:30.277128 1944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:30.277226 1944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:30.277390 1944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:30.286761 1944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab
--principal=oryx/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:30.288090 1944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:30.289529 1944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:30.302546 1950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:30.302577 1951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:30.302490 1953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:30.303259 1944 server_base.cc:1034] running on GCE node
I20260807 18:28:30.303951 1944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:30.304962 1944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:30.306192 1944 hybrid_clock.cc:648] HybridClock initialized: now 1786127310306152 us; error 50 us; skew 500 ppm
I20260807 18:28:30.306524 1944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:30 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127310, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:30.309741 1944 init.cc:377] Logged in from keytab as oryx/127.27.1.194@KRBTEST.COM (short username oryx)
I20260807 18:28:30.311182 1944 webserver.cc:460] Webserver started at http://127.27.1.194:33173/ using document root <none> and password file <none>
I20260807 18:28:30.312134 1944 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:30.312291 1944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:30.312644 1944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:30.316182 1944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "698923a8cfda4e21a16dbf1b1239f5bf"
format_stamp: "Formatted at 2026-08-07 18:28:30 on dist-test-slave-1xrh"
server_key: "14103f01284fe70d7afa5b26b39aff78"
server_key_iv: "7a7636968a11168afbac123da1437981"
server_key_version: "encryptionkey@0"
I20260807 18:28:30.317028 1944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "698923a8cfda4e21a16dbf1b1239f5bf"
format_stamp: "Formatted at 2026-08-07 18:28:30 on dist-test-slave-1xrh"
server_key: "14103f01284fe70d7afa5b26b39aff78"
server_key_iv: "7a7636968a11168afbac123da1437981"
server_key_version: "encryptionkey@0"
I20260807 18:28:30.322461 1944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 18:28:30.325867 1963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:30.327394 1944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:30.327580 1944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "698923a8cfda4e21a16dbf1b1239f5bf"
format_stamp: "Formatted at 2026-08-07 18:28:30 on dist-test-slave-1xrh"
server_key: "14103f01284fe70d7afa5b26b39aff78"
server_key_iv: "7a7636968a11168afbac123da1437981"
server_key_version: "encryptionkey@0"
I20260807 18:28:30.327768 1944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:30.349968 1944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:30.353935 1944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:30.354282 1944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:30.355345 1944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:30.356657 1944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:30.356732 1944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:30.356902 1944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:30.356964 1944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:30.390825 1944 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:39689
I20260807 18:28:30.390909 2081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:39689 every 8 connection(s)
I20260807 18:28:30.392115 1944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:30.395226 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 1944
I20260807 18:28:30.395408 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:28:30.396023 27655 external_mini_cluster.cc:1428] Setting key 3e3a152b0265cd2750d0710c99b0d552
Aug 07 18:28:30 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127310, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:30.414444 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:30.394630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:49223 (local address 127.27.1.254:38033)
0807 18:28:30.394805 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:30.394810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:30.396140 (+ 1330us) server_negotiation.cc:407] Connection header received
0807 18:28:30.398316 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:30.398320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:30.398385 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:30.398465 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:30.400570 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:30.401247 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:30.403656 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:30.404074 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:30.404166 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:30.408295 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:30.408300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:30.408303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:30.408336 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:30.410928 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:30.411950 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:30.411955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:30.411959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:30.412152 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:30.412591 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:30.412596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:30.412615 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:30.412970 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:30.413137 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:30.413807 (+ 670us) server_negotiation.cc:299] Negotiation successful
0807 18:28:30.414041 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 18:28:30.415477 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:30.395045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.194:49223)
0807 18:28:30.395956 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:30.396007 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:30.398032 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:30.398766 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:30.398777 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:30.399564 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:30.400374 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:30.400407 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:30.401395 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:30.401402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:30.403518 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:30.403536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:30.404293 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:30.404302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:30.404626 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:30.405358 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:30.405394 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:30.408060 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:30.411167 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:30.411176 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:30.411194 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:30.411785 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:30.412282 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:30.412285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:30.412287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:30.412449 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:30.413105 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:30.413112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:30.413443 (+ 331us) client_negotiation.cc:769] Sending connection context
0807 18:28:30.413694 (+ 251us) client_negotiation.cc:240] Negotiation successful
0807 18:28:30.414244 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":323,"threads_started":1}
I20260807 18:28:30.416884 2082 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38033
I20260807 18:28:30.417194 2082 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:30.417906 2082 heartbeater.cc:511] Master 127.27.1.254:38033 requested a full tablet report, sending...
I20260807 18:28:30.419410 1732 ts_manager.cc:194] Registered new tserver with Master: 698923a8cfda4e21a16dbf1b1239f5bf (127.27.1.194:39689)
I20260807 18:28:30.420650 1732 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:49223
WARNING: no policy specified for oryx/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab.
Entry for principal oryx/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab.
I20260807 18:28:30.452267 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38033
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab
--principal=oryx/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:30.784240 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:30.784487 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:30.784581 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:30.791781 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:30.791884 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:30.792013 2089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:30.801713 2089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab
--principal=oryx/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:30.802877 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:30.804312 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:30.817684 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:30.817649 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:30.817777 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:30.818804 2089 server_base.cc:1034] running on GCE node
I20260807 18:28:30.819450 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:30.820328 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:30.821568 2089 hybrid_clock.cc:648] HybridClock initialized: now 1786127310821529 us; error 28 us; skew 500 ppm
I20260807 18:28:30.821827 2089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:30 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127310, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:30.825066 2089 init.cc:377] Logged in from keytab as oryx/127.27.1.195@KRBTEST.COM (short username oryx)
I20260807 18:28:30.826468 2089 webserver.cc:460] Webserver started at http://127.27.1.195:44021/ using document root <none> and password file <none>
I20260807 18:28:30.827116 2089 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:30.827306 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:30.827623 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:30.830375 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "3eb0a9e28caf420487cc3e2db20f96c3"
format_stamp: "Formatted at 2026-08-07 18:28:30 on dist-test-slave-1xrh"
server_key: "495f7a271843426048e04bc054e07641"
server_key_iv: "3fca96b9856aff0a569e35f285c1cd2b"
server_key_version: "encryptionkey@0"
I20260807 18:28:30.831285 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "3eb0a9e28caf420487cc3e2db20f96c3"
format_stamp: "Formatted at 2026-08-07 18:28:30 on dist-test-slave-1xrh"
server_key: "495f7a271843426048e04bc054e07641"
server_key_iv: "3fca96b9856aff0a569e35f285c1cd2b"
server_key_version: "encryptionkey@0"
I20260807 18:28:30.836247 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 18:28:30.839658 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:30.840942 2089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 18:28:30.841099 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "3eb0a9e28caf420487cc3e2db20f96c3"
format_stamp: "Formatted at 2026-08-07 18:28:30 on dist-test-slave-1xrh"
server_key: "495f7a271843426048e04bc054e07641"
server_key_iv: "3fca96b9856aff0a569e35f285c1cd2b"
server_key_version: "encryptionkey@0"
I20260807 18:28:30.841295 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:30.856711 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:30.860671 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:30.860985 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:30.861945 2089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:30.863720 2089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:30.863826 2089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:30.864001 2089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:30.864077 2089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:30.898555 2089 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:34445
I20260807 18:28:30.898602 2225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:34445 every 8 connection(s)
I20260807 18:28:30.899874 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:30.905853 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 2089
I20260807 18:28:30.906051 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:28:30.906654 27655 external_mini_cluster.cc:1428] Setting key 6375500d3269684a62ca61ea7eca5c6b
I20260807 18:28:30.906948 1938 heartbeater.cc:503] Master 127.27.1.254:38033 was elected leader, sending a full tablet report...
Aug 07 18:28:30 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127310, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:30.919915 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:30.902346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:50573 (local address 127.27.1.254:38033)
0807 18:28:30.902559 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:30.902565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:30.903733 (+ 1168us) server_negotiation.cc:407] Connection header received
0807 18:28:30.905428 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:30.905432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:30.905503 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:30.905592 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:30.907650 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:30.908868 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:30.910624 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:30.911027 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:30.911178 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:30.914584 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:30.914589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:30.914592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:30.914624 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:30.917277 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:30.917924 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:30.917928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:30.917930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:30.918061 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:30.918420 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:30.918423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:30.918425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:30.918696 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:30.918818 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:30.919489 (+ 671us) server_negotiation.cc:299] Negotiation successful
0807 18:28:30.919668 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 18:28:30.921187 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:30.902828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.195:50573)
0807 18:28:30.903584 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:30.903621 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:30.905190 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:30.905876 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:30.905888 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:30.906625 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:30.907457 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:30.907486 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:30.909065 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:30.909069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:30.910494 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:30.910506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:30.911266 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:30.911271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:30.911489 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:30.912234 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:30.912255 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:30.914394 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:30.917399 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:30.917406 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:30.917417 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:30.917817 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:30.918152 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:30.918155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:30.918157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:30.918323 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:30.918809 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:30.918814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:30.919099 (+ 285us) client_negotiation.cc:769] Sending connection context
0807 18:28:30.919380 (+ 281us) client_negotiation.cc:240] Negotiation successful
0807 18:28:30.919737 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":324,"threads_started":1}
I20260807 18:28:30.922631 2226 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38033
I20260807 18:28:30.922930 2226 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:30.923707 2226 heartbeater.cc:511] Master 127.27.1.254:38033 requested a full tablet report, sending...
I20260807 18:28:30.925055 1732 ts_manager.cc:194] Registered new tserver with Master: 3eb0a9e28caf420487cc3e2db20f96c3 (127.27.1.195:34445)
I20260807 18:28:30.925846 1732 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:50573
I20260807 18:28:30.934190 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.1.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.253.keytab.
Entry for principal oryx/127.27.1.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.253.keytab.
I20260807 18:28:30.989116 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.253:37827
--webserver_interface=127.27.1.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.253.keytab
--principal=oryx/127.27.1.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:38033,127.27.1.253:37827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:31.342736 2232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:31.343084 2232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:31.343261 2232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:31.350605 2232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:31.350755 2232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:31.350822 2232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:31.350883 2232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:31.350927 2232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:31.360630 2232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:38033,127.27.1.253:37827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.253.keytab
--principal=oryx/127.27.1.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.253:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.1.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
W20260807 18:28:31.361068 2232 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.1.254:38033,127.27.1.253:37827'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 18:28:31.362243 2232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:31.363878 2232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:31.375841 2237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:31.375891 2240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:31.375933 2238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:31.376737 2232 server_base.cc:1034] running on GCE node
I20260807 18:28:31.377431 2232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:31.378698 2232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:31.379922 2232 hybrid_clock.cc:648] HybridClock initialized: now 1786127311379852 us; error 45 us; skew 500 ppm
I20260807 18:28:31.380265 2232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:31 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127311, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:31.383482 2232 init.cc:377] Logged in from keytab as oryx/127.27.1.253@KRBTEST.COM (short username oryx)
I20260807 18:28:31.384945 2232 webserver.cc:460] Webserver started at http://127.27.1.253:38741/ using document root <none> and password file <none>
I20260807 18:28:31.385649 2232 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:31.385782 2232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:31.386106 2232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:31.389284 2232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data/instance:
uuid: "eec35020298c422d8dfbbea46d5dd5cc"
format_stamp: "Formatted at 2026-08-07 18:28:31 on dist-test-slave-1xrh"
server_key: "e8a36a8adbb7d4fa03c5842124bc2a2c"
server_key_iv: "7ab3e6d9d74463dc67f63b3da1651394"
server_key_version: "encryptionkey@0"
I20260807 18:28:31.390129 2232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal/instance:
uuid: "eec35020298c422d8dfbbea46d5dd5cc"
format_stamp: "Formatted at 2026-08-07 18:28:31 on dist-test-slave-1xrh"
server_key: "e8a36a8adbb7d4fa03c5842124bc2a2c"
server_key_iv: "7ab3e6d9d74463dc67f63b3da1651394"
server_key_version: "encryptionkey@0"
I20260807 18:28:31.396004 2232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260807 18:28:31.399942 2248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:31.401341 2232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:28:31.401572 2232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal
uuid: "eec35020298c422d8dfbbea46d5dd5cc"
format_stamp: "Formatted at 2026-08-07 18:28:31 on dist-test-slave-1xrh"
server_key: "e8a36a8adbb7d4fa03c5842124bc2a2c"
server_key_iv: "7ab3e6d9d74463dc67f63b3da1651394"
server_key_version: "encryptionkey@0"
I20260807 18:28:31.401809 2232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:31.423166 2082 heartbeater.cc:503] Master 127.27.1.254:38033 was elected leader, sending a full tablet report...
I20260807 18:28:31.436760 2232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:31.440876 2232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:31.441092 2232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:31.460767 2232 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.253:37827
I20260807 18:28:31.460800 2303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.253:37827 every 8 connection(s)
I20260807 18:28:31.462356 2232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
I20260807 18:28:31.462968 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 2232
I20260807 18:28:31.466746 2304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:31.475267 2304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:28:31 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.253@KRBTEST.COM
Aug 07 18:28:31 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127311, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.253@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:31.487620 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.465401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.27.1.253:37827)
0807 18:28:31.466218 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:31.466230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:31.466270 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:28:31.468020 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:31.468042 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:31.468398 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:31.468698 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:31.470571 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.472061 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.473844 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.474457 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.474582 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.480008 (+ 5426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:31.480017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:31.480035 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:31.480074 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:31.483462 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.484269 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.484278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.484285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.484481 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.484910 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.484917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.484920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.485414 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:31.485742 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:31.486206 (+ 464us) server_negotiation.cc:299] Negotiation successful
0807 18:28:31.486605 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":379,"threads_started":1}
I20260807 18:28:31.493673 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.478411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46280)
0807 18:28:31.479280 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:31.479332 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:31.479483 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:31.480238 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:31.480247 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:31.480859 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:31.481139 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.481147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.482582 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.482586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.483927 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.483944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.484886 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.484895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.485031 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.485653 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:31.485670 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:31.488027 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:31.491528 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.491537 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.491541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.491919 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.492242 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.492245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.492247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.492378 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.492877 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:31.492884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:31.493020 (+ 136us) client_negotiation.cc:769] Sending connection context
0807 18:28:31.493179 (+ 159us) client_negotiation.cc:240] Negotiation successful
0807 18:28:31.493399 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":473,"threads_started":1}
I20260807 18:28:31.493680 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.478442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.27.1.254:38033)
0807 18:28:31.479864 (+ 1422us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:31.479870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:31.479888 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:28:31.479949 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:31.479957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:31.480055 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:31.480181 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:31.481297 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.482426 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.484079 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.484695 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.484785 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.488213 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:31.488218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:31.488221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:31.488254 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:31.491412 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.492029 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.492033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.492035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.492154 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.492472 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.492475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.492477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.492773 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:31.492883 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:31.493274 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 18:28:31.493436 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1291,"thread_start_us":1206,"threads_started":1}
I20260807 18:28:31.495316 2304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:28:31 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127311, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.253@KRBTEST.COM for oryx/127.27.1.253@KRBTEST.COM
I20260807 18:28:31.507504 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.495984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:37827 (local address 127.0.0.1:39372)
0807 18:28:31.496185 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:31.496205 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:31.496297 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:31.496579 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:31.496583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:31.497001 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:31.497119 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.497126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.498080 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.498085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.498977 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.498989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.499695 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.499700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.499914 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.500440 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:31.500455 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:31.502261 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:31.505057 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.505064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.505068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.505520 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.505898 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.505903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.505908 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.506030 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.506521 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:31.506526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:31.506662 (+ 136us) client_negotiation.cc:769] Sending connection context
0807 18:28:31.506793 (+ 131us) client_negotiation.cc:240] Negotiation successful
0807 18:28:31.507078 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260807 18:28:31.507581 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.496134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.27.1.253:37827)
0807 18:28:31.496291 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:31.496296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:31.496313 (+ 17us) server_negotiation.cc:407] Connection header received
0807 18:28:31.496401 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:31.496405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:31.496467 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:31.496564 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:31.497245 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.497938 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.499116 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.499553 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.499685 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.502423 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:31.502429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:31.502433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:31.502470 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:31.504919 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.505629 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.505636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.505640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.505774 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.506130 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.506137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.506140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.506387 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:31.506551 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:31.506863 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 18:28:31.507244 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":189,"server-negotiator.queue_time_us":43}
I20260807 18:28:31.512307 2304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: Bootstrap starting.
I20260807 18:28:31.515575 2304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:31.516605 2304 log.cc:824] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:31.519202 2304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: No bootstrap required, opened a new log
I20260807 18:28:31.525959 2304 raft_consensus.cc:348] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } }
I20260807 18:28:31.526327 2304 raft_consensus.cc:374] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:31.526427 2304 raft_consensus.cc:729] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec35020298c422d8dfbbea46d5dd5cc, State: Initialized, Role: FOLLOWER
I20260807 18:28:31.526988 2304 consensus_queue.cc:260] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } }
I20260807 18:28:31.527905 2311 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } } }
I20260807 18:28:31.528237 2311 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:28:31.528848 2304 sys_catalog.cc:564] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: configured and running, proceeding with master startup.
W20260807 18:28:31.537561 2322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 18:28:31.537701 2322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 18:28:31.540911 2232 master_runner.cc:428] Detected that this master eec35020298c422d8dfbbea46d5dd5cc is joining an existing cluster
I20260807 18:28:31.541014 2232 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.1.253:37827
I20260807 18:28:31.570046 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.545145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.27.1.254:38033)
0807 18:28:31.545351 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:31.545360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:31.553280 (+ 7920us) server_negotiation.cc:407] Connection header received
0807 18:28:31.553508 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:31.553517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:31.553661 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:31.553798 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:31.555324 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.556471 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.558687 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.559407 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.559521 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.562600 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:31.562606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:31.562610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:31.562649 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:31.566602 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.567565 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.567571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.567574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.567734 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.568207 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.568213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.568217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.568621 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:31.568752 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:31.569410 (+ 658us) server_negotiation.cc:299] Negotiation successful
0807 18:28:31.569670 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 18:28:31.570708 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.547265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.27.1.253:37827)
0807 18:28:31.547467 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:31.547474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:31.554023 (+ 6549us) server_negotiation.cc:407] Connection header received
0807 18:28:31.554164 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:31.554170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:31.554275 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:31.554385 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:31.555773 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.556800 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.558682 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.559424 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.559521 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.562579 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:31.562587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:31.562591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:31.562628 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:31.567599 (+ 4971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.568396 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.568400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.568408 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.568558 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.569004 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.569013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.569016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.569410 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:31.569586 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:31.570119 (+ 533us) server_negotiation.cc:299] Negotiation successful
0807 18:28:31.570332 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"spinlock_wait_cycles":46336}
I20260807 18:28:31.571393 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.545330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46288)
0807 18:28:31.553159 (+ 7829us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:31.553184 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:31.553343 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:31.553891 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:31.553896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:31.554618 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:31.555082 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.555097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.556652 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.556657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.558517 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.558533 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.559876 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.559883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.560125 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.561021 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:31.561055 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:31.562399 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:31.566765 (+ 4366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.566770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.566775 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.567411 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.567885 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.567891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.567894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.568062 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.568830 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:31.568834 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:31.569057 (+ 223us) client_negotiation.cc:769] Sending connection context
0807 18:28:31.570321 (+ 1264us) client_negotiation.cc:240] Negotiation successful
0807 18:28:31.570608 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7680,"mutex_wait_us":147,"spinlock_wait_cycles":33664,"thread_start_us":7546,"threads_started":1}
I20260807 18:28:31.571988 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.545345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:37827 (local address 127.0.0.1:39374)
0807 18:28:31.553918 (+ 8573us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:31.553936 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:31.554060 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:31.554527 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:31.554531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:31.555314 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:31.555630 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.555641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.557009 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.557019 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.558557 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.558569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.559844 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.559857 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.560124 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.561169 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:31.561191 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:31.562398 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:31.567727 (+ 5329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.567731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.567734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.568282 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.568699 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.568703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.568706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.568874 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.569681 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:31.569685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:31.569845 (+ 160us) client_negotiation.cc:769] Sending connection context
0807 18:28:31.570991 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0807 18:28:31.571308 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8448,"mutex_wait_us":118,"spinlock_wait_cycles":1920,"thread_start_us":8243,"threads_started":1}
WARNING: no policy specified for oryx/127.27.1.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.252@KRBTEST.COM" created.
Aug 07 18:28:31 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127309, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.254@KRBTEST.COM for oryx/127.27.1.253@KRBTEST.COM
I20260807 18:28:31.600385 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.578128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.27.1.253:37827)
0807 18:28:31.578361 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:31.578370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:31.585817 (+ 7447us) server_negotiation.cc:407] Connection header received
0807 18:28:31.586196 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:31.586200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:31.586282 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:31.586393 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:31.587647 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.588509 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.589955 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.590422 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.590587 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.593922 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:31.593927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:31.593930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:31.593960 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:31.597471 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.598145 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.598151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.598154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.598285 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.598674 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.598679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.598682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.599084 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:31.599272 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:31.599743 (+ 471us) server_negotiation.cc:299] Negotiation successful
0807 18:28:31.599960 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:28:31.600440 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.578094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:37827 (local address 127.0.0.1:39378)
0807 18:28:31.585683 (+ 7589us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:31.585720 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:31.585947 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:31.586459 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:31.586466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:31.587270 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:31.587492 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.587502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.588659 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.588664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.589813 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.589826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.590604 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.590610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.590816 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.591544 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:31.591566 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:31.593726 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:31.597595 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.597607 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.597615 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.598016 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.598403 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.598408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.598411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.598547 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.599256 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:31.599264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:31.599452 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 18:28:31.599635 (+ 183us) client_negotiation.cc:240] Negotiation successful
0807 18:28:31.599960 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7379,"thread_start_us":7219,"threads_started":1}
I20260807 18:28:31.602617 1732 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.1.253:37827: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc"
member_type: NON_VOTER
last_known_addr {
host: "127.27.1.253"
port: 37827
}
attrs {
promote: true
}
}
dest_uuid: "c98333801763413d99a0d26660b81d1a"
cas_config_opid_index: -1
I20260807 18:28:31.603740 1732 consensus_queue.cc:237] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } }
I20260807 18:28:31.608120 2338 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER eec35020298c422d8dfbbea46d5dd5cc (127.27.1.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } } }
I20260807 18:28:31.609257 2277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:31.612967 2338 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.1.253:37827
I20260807 18:28:31.613327 2277 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Refusing update from remote peer c98333801763413d99a0d26660b81d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 18:28:31.614112 2232 master.cc:537] Master@127.27.1.253:37827 shutting down...
I20260807 18:28:31.614404 2338 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.252.keytab.
Entry for principal oryx/127.27.1.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.252.keytab.
I20260807 18:28:31.617056 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.252:33443
--webserver_interface=127.27.1.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.252.keytab
--principal=oryx/127.27.1.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:38033,127.27.1.253:37827,127.27.1.252:33443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:31.617460 1717 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260807 18:28:31.625162 2337 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } } }
I20260807 18:28:31.625689 2337 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:31.629916 2232 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:31.630471 2232 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:31.630718 2232 tablet_replica.cc:330] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: stopping tablet replica
W20260807 18:28:31.633250 1717 consensus_peers.cc:487] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a -> Peer eec35020298c422d8dfbbea46d5dd5cc (127.27.1.253:37827): Couldn't send request to peer eec35020298c422d8dfbbea46d5dd5cc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 18:28:31.638579 2232 master.cc:552] Master@127.27.1.253:37827 shutdown complete.
I20260807 18:28:31.638700 2232 master_runner.cc:305] Clearing existing system tablet
I20260807 18:28:31.639181 2232 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:31.639279 2232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:31.642760 2232 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 18:28:31.644613 2343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:31.644973 2232 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:31.645411 2232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260807 18:28:31.645533 2232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal
uuid: "eec35020298c422d8dfbbea46d5dd5cc"
format_stamp: "Formatted at 2026-08-07 18:28:31 on dist-test-slave-1xrh"
server_key: "e8a36a8adbb7d4fa03c5842124bc2a2c"
server_key_iv: "7ab3e6d9d74463dc67f63b3da1651394"
server_key_version: "encryptionkey@0"
I20260807 18:28:31.650453 2232 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:31.657249 2232 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 18:28:31.657465 2232 log.cc:1196] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 18:28:31.657928 2232 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: Deleting consensus metadata
I20260807 18:28:31.658254 2232 master_runner.cc:315] Copying system tablet from 127.27.1.254:38033
I20260807 18:28:31.660770 2232 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.1.254:38033
I20260807 18:28:31.677184 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.661807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.27.1.254:38033)
0807 18:28:31.662068 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:31.662075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:31.662518 (+ 443us) server_negotiation.cc:407] Connection header received
0807 18:28:31.662791 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:31.662797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:31.662880 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:31.662953 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:31.664206 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.665383 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.667342 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.667982 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.668080 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.670551 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:31.670558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:31.670563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:31.670602 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:31.674063 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.674906 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.674912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.674916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.675080 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:31.675538 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:31.675543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:31.675547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:31.675912 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:31.676069 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:31.676551 (+ 482us) server_negotiation.cc:299] Negotiation successful
0807 18:28:31.676797 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:28:31.677374 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:31.661562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46300)
0807 18:28:31.662410 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:31.662431 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:31.662630 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:31.663199 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:31.663204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:31.663715 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:31.664029 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.664041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.665563 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.665570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.667078 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:31.667094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:31.668361 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:31.668367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:31.668562 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:31.669270 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:31.669300 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:31.670324 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:31.674225 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.674238 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.674242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.674756 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.675233 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:31.675238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:31.675242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:31.675416 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:31.676055 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:31.676060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:31.676256 (+ 196us) client_negotiation.cc:769] Sending connection context
0807 18:28:31.676435 (+ 179us) client_negotiation.cc:240] Negotiation successful
0807 18:28:31.676772 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":605,"threads_started":1}
I20260807 18:28:31.678424 1757 tablet_copy_service.cc:140] P c98333801763413d99a0d26660b81d1a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer eec35020298c422d8dfbbea46d5dd5cc ({username='oryx', principal='oryx/127.27.1.253@KRBTEST.COM'} at 127.0.0.1:46300)
I20260807 18:28:31.678750 1757 tablet_copy_service.cc:161] P c98333801763413d99a0d26660b81d1a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer eec35020298c422d8dfbbea46d5dd5cc at {username='oryx', principal='oryx/127.27.1.253@KRBTEST.COM'} at 127.0.0.1:46300: session id = eec35020298c422d8dfbbea46d5dd5cc-00000000000000000000000000000000
I20260807 18:28:31.683260 1757 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 18:28:31.685623 2232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 18:28:31.692579 2232 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: tablet copy: Starting download of 0 data blocks...
I20260807 18:28:31.692879 2232 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: tablet copy: Starting download of 1 WAL segments...
I20260807 18:28:31.695855 2232 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 18:28:31.700341 1757 tablet_copy_service.cc:342] P c98333801763413d99a0d26660b81d1a: Request end of tablet copy session eec35020298c422d8dfbbea46d5dd5cc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.1.253@KRBTEST.COM'} at 127.0.0.1:46300
I20260807 18:28:31.700556 1757 tablet_copy_service.cc:433] P c98333801763413d99a0d26660b81d1a: ending tablet copy session eec35020298c422d8dfbbea46d5dd5cc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer eec35020298c422d8dfbbea46d5dd5cc
I20260807 18:28:31.710742 2232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:31.715497 2351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:31.717150 2352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:31.721674 2232 server_base.cc:1034] running on GCE node
W20260807 18:28:31.722045 2354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:31.722442 2232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:31.723064 2232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:31.724324 2232 hybrid_clock.cc:648] HybridClock initialized: now 1786127311724263 us; error 52 us; skew 500 ppm
I20260807 18:28:31.724574 2232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:31 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127311, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:31.726471 2232 init.cc:377] Logged in from keytab as oryx/127.27.1.253@KRBTEST.COM (short username oryx)
I20260807 18:28:31.727837 2232 webserver.cc:460] Webserver started at http://127.27.1.253:44353/ using document root <none> and password file <none>
I20260807 18:28:31.728170 2232 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:31.728271 2232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:31.730759 2232 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 18:28:31.732635 2362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:31.733057 2232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 18:28:31.733219 2232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal
uuid: "eec35020298c422d8dfbbea46d5dd5cc"
format_stamp: "Formatted at 2026-08-07 18:28:31 on dist-test-slave-1xrh"
server_key: "e8a36a8adbb7d4fa03c5842124bc2a2c"
server_key_iv: "7ab3e6d9d74463dc67f63b3da1651394"
server_key_version: "encryptionkey@0"
I20260807 18:28:31.733397 2232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:31.742274 2232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:31.745787 2232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:31.782701 2232 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.253:37827
I20260807 18:28:31.782742 2419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.253:37827 every 8 connection(s)
I20260807 18:28:31.783715 2232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
I20260807 18:28:31.790879 2420 sys_catalog.cc:263] Verifying existing consensus state
I20260807 18:28:31.793416 2420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: Bootstrap starting.
I20260807 18:28:31.806371 2420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 18:28:31.806864 2420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: Bootstrap complete.
I20260807 18:28:31.807770 2420 raft_consensus.cc:348] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } }
I20260807 18:28:31.808022 2420 raft_consensus.cc:729] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eec35020298c422d8dfbbea46d5dd5cc, State: Initialized, Role: LEARNER
I20260807 18:28:31.808300 2420 consensus_queue.cc:260] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } }
I20260807 18:28:31.809238 2423 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } } }
I20260807 18:28:31.809545 2420 sys_catalog.cc:564] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:31.809597 2423 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: This master's current role is: LEARNER
I20260807 18:28:31.821338 2434 catalog_manager.cc:1259] Loaded cluster ID: 1bcd0fbcc44d441687caedeef987db55
I20260807 18:28:31.821452 2434 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: loading cluster ID for follower catalog manager: success
I20260807 18:28:31.823875 2434 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: acquiring CA information for follower catalog manager: success
I20260807 18:28:31.825632 2434 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 18:28:31.928277 2226 heartbeater.cc:503] Master 127.27.1.254:38033 was elected leader, sending a full tablet report...
W20260807 18:28:32.058534 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:32.058861 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:32.059012 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:32.066527 2341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:32.066663 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:32.066764 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:32.066870 2341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:32.066951 2341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:32.076634 2341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:38033,127.27.1.253:37827,127.27.1.252:33443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.252.keytab
--principal=oryx/127.27.1.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.252:33443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.1.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:32.077960 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:32.079299 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:32.095225 2440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:32.095643 2441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:32.096426 2443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:32.097812 2341 server_base.cc:1034] running on GCE node
I20260807 18:28:32.098814 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:32.100474 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:32.101708 2341 hybrid_clock.cc:648] HybridClock initialized: now 1786127312101649 us; error 53 us; skew 500 ppm
I20260807 18:28:32.102032 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.092723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:37827 (local address 127.0.0.1:39388)
0807 18:28:32.092912 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:32.092934 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:32.093092 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:32.094490 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:32.094495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:32.094532 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:32.094739 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.094759 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.097135 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.097141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.099446 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.099463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.101290 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.101296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.101451 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.101493 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 18:28:32.101575 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 18:28:32.101738 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 18:28:32.102147 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.092921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.27.1.253:37827)
0807 18:28:32.094052 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.094058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.094087 (+ 29us) server_negotiation.cc:407] Connection header received
0807 18:28:32.094187 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.094196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.094284 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.094409 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:32.095007 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.096980 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.099630 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.101071 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.101228 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.101463 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.101630 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.101775 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":803,"threads_started":1}
I20260807 18:28:32.102495 2341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:28:32.103410 2393 raft_consensus.cc:1203] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260807 18:28:32.104086 2423 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c98333801763413d99a0d26660b81d1a. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } } }
I20260807 18:28:32.104360 2423 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: This master's current role is: LEARNER
Aug 07 18:28:32 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127312, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:32.106415 2341 init.cc:377] Logged in from keytab as oryx/127.27.1.252@KRBTEST.COM (short username oryx)
I20260807 18:28:32.108242 2341 webserver.cc:460] Webserver started at http://127.27.1.252:36383/ using document root <none> and password file <none>
I20260807 18:28:32.109227 2341 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:32.109408 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:32.109766 2341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:32.113500 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data/instance:
uuid: "9bb886b775c34a0d964e7dd52eb2518d"
format_stamp: "Formatted at 2026-08-07 18:28:32 on dist-test-slave-1xrh"
server_key: "becc20323fbfb7cbe06a32bbd4bf1df0"
server_key_iv: "34be01b0a7e7acbaf51022fd983e4cf9"
server_key_version: "encryptionkey@0"
I20260807 18:28:32.114269 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal/instance:
uuid: "9bb886b775c34a0d964e7dd52eb2518d"
format_stamp: "Formatted at 2026-08-07 18:28:32 on dist-test-slave-1xrh"
server_key: "becc20323fbfb7cbe06a32bbd4bf1df0"
server_key_iv: "34be01b0a7e7acbaf51022fd983e4cf9"
server_key_version: "encryptionkey@0"
I20260807 18:28:32.119812 2341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 18:28:32.123381 2452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:32.124733 2341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:28:32.124953 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal
uuid: "9bb886b775c34a0d964e7dd52eb2518d"
format_stamp: "Formatted at 2026-08-07 18:28:32 on dist-test-slave-1xrh"
server_key: "becc20323fbfb7cbe06a32bbd4bf1df0"
server_key_iv: "34be01b0a7e7acbaf51022fd983e4cf9"
server_key_version: "encryptionkey@0"
I20260807 18:28:32.125157 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:32.141856 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:32.146380 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:32.146680 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:32.167802 2341 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.252:33443
I20260807 18:28:32.167837 2509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.252:33443 every 8 connection(s)
I20260807 18:28:32.169246 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
I20260807 18:28:32.173149 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:32.174991 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 2341
I20260807 18:28:32.182678 2510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:28:32 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.252@KRBTEST.COM
Aug 07 18:28:32 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127312, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.252@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:32.201032 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.177429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.27.1.252:33443)
0807 18:28:32.178417 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.178428 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.178469 (+ 41us) server_negotiation.cc:407] Connection header received
0807 18:28:32.180105 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.180130 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.180505 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.180848 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:32.182810 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.184356 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.185997 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.187139 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.187315 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.192959 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:32.192968 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:32.192983 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:32.193031 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:32.196867 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.197587 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.197599 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.197609 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.197863 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.198235 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.198241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.198244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.198735 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:32.199046 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.199561 (+ 515us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.199935 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":541,"threads_started":1}
I20260807 18:28:32.203756 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.185522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.27.1.254:38033)
0807 18:28:32.186841 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.186848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.186867 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:28:32.186927 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.186932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.187023 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.187175 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:32.188770 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.189678 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.191924 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.192621 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.192751 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.197260 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:32.197265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:32.197269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:32.197303 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:32.200680 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.201571 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.201577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.201582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.201744 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.202141 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.202147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.202151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.202565 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:32.202704 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.203303 (+ 599us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.203483 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1196,"thread_start_us":1062,"threads_started":1}
I20260807 18:28:32.203871 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.185323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46306)
0807 18:28:32.186224 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:32.186263 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:32.186395 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:32.187314 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:32.187321 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:32.188199 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:32.188600 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.188619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.189849 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.189857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.191750 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.191763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.192955 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.192964 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.193198 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.193927 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:32.193953 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:32.196996 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:32.200825 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.200834 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.200840 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.201421 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.201872 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.201874 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.201877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.202017 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.202711 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:32.202718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:32.202906 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 18:28:32.203164 (+ 258us) client_negotiation.cc:240] Negotiation successful
0807 18:28:32.203453 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":416,"threads_started":1}
I20260807 18:28:32.205180 2510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:28:32 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127312, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.252@KRBTEST.COM for oryx/127.27.1.253@KRBTEST.COM
I20260807 18:28:32.218034 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.205762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:37827 (local address 127.0.0.1:39392)
0807 18:28:32.205926 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:32.205941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:32.206024 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:32.206471 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:32.206475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:32.206818 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:32.206942 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.206949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.207995 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.208001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.208903 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.208913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.209741 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.209747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.209938 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.210553 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:32.210619 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:32.212598 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:32.215851 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.215854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.215856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.216300 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.216679 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.216683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.216685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.216807 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.217355 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:32.217359 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:32.217519 (+ 160us) client_negotiation.cc:769] Sending connection context
0807 18:28:32.217629 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 18:28:32.217817 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 18:28:32.218454 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.206012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.27.1.253:37827)
0807 18:28:32.206214 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.206219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.206234 (+ 15us) server_negotiation.cc:407] Connection header received
0807 18:28:32.206281 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.206285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.206342 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.206451 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:32.207100 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.207858 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.209048 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.209606 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.209735 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.212801 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:32.212806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:32.212809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:32.212891 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:32.215728 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.216412 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.216416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.216418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.216543 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.216933 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.216937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.216939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.217257 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:32.217370 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.217756 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.218006 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 18:28:32.219458 2510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:28:32 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127312, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.252@KRBTEST.COM for oryx/127.27.1.252@KRBTEST.COM
I20260807 18:28:32.232026 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.220196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33443 (local address 127.0.0.1:51314)
0807 18:28:32.220445 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:32.220459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:32.220545 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:32.220785 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:32.220788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:32.221083 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:32.221210 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.221217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.222041 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.222044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.222927 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.222939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.223611 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.223633 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.223812 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.224366 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:32.224382 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:32.226327 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:32.229398 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.229402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.229425 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.230028 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.230345 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.230348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.230350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.230501 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.230972 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:32.230977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:32.231117 (+ 140us) client_negotiation.cc:769] Sending connection context
0807 18:28:32.231303 (+ 186us) client_negotiation.cc:240] Negotiation successful
0807 18:28:32.231595 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 18:28:32.232059 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.220177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.27.1.252:33443)
0807 18:28:32.220370 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.220376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.220522 (+ 146us) server_negotiation.cc:407] Connection header received
0807 18:28:32.220628 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.220631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.220682 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.220753 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:32.221318 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.221937 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.223047 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.223488 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.223585 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.226482 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:32.226488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:32.226492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:32.226518 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:32.229195 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.230129 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.230133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.230135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.230238 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.230596 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.230598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.230600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.230856 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:32.231006 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.231385 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.231730 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":57}
I20260807 18:28:32.237200 2510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Bootstrap starting.
I20260807 18:28:32.240638 2510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:32.241677 2510 log.cc:824] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:32.244515 2510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: No bootstrap required, opened a new log
I20260807 18:28:32.252089 2510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } }
I20260807 18:28:32.252439 2510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:32.252570 2510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb886b775c34a0d964e7dd52eb2518d, State: Initialized, Role: FOLLOWER
I20260807 18:28:32.253118 2510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } }
I20260807 18:28:32.254213 2518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } } }
I20260807 18:28:32.254586 2518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:28:32.255419 2510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: configured and running, proceeding with master startup.
W20260807 18:28:32.265663 2530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 18:28:32.265808 2530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 18:28:32.270190 2341 master_runner.cc:428] Detected that this master 9bb886b775c34a0d964e7dd52eb2518d is joining an existing cluster
I20260807 18:28:32.270351 2341 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.1.252:33443
I20260807 18:28:32.303113 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.274401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.27.1.254:38033)
0807 18:28:32.274584 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.274591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.286533 (+ 11942us) server_negotiation.cc:407] Connection header received
0807 18:28:32.286777 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.286783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.286863 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.286955 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:32.288970 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.290161 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.292735 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.293402 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.293518 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.296259 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:32.296266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:32.296273 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:32.296321 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:32.299945 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.300825 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.300830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.300833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.300974 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.301444 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.301449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.301452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.301797 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:32.301922 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.302573 (+ 651us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.302795 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 18:28:32.303174 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.274608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.27.1.252:33443)
0807 18:28:32.274804 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.274810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.281921 (+ 7111us) server_negotiation.cc:407] Connection header received
0807 18:28:32.282122 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.282129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.282220 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.282328 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:32.287668 (+ 5340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.289019 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.291574 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.292311 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.292423 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.295616 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:32.295622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:32.295627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:32.295664 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:32.299695 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.300816 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.300821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.300824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.300955 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.301430 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.301434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.301437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.301789 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:32.301934 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.302539 (+ 605us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.302756 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":59}
I20260807 18:28:32.304173 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.274606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33443 (local address 127.0.0.1:51316)
0807 18:28:32.281799 (+ 7193us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:32.281820 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:32.282001 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:32.282531 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:32.282538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:32.285031 (+ 2493us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:32.287489 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.287505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.289240 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.289248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.291410 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.291425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.292891 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.292898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.293139 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.293900 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:32.293925 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:32.295428 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:32.300119 (+ 4691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.300123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.300126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.300691 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.301106 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.301111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.301114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.301315 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.302009 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:32.302012 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:32.302282 (+ 270us) client_negotiation.cc:769] Sending connection context
0807 18:28:32.303420 (+ 1138us) client_negotiation.cc:240] Negotiation successful
0807 18:28:32.303662 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7053,"mutex_wait_us":89,"spinlock_wait_cycles":65280,"thread_start_us":6886,"threads_started":1}
I20260807 18:28:32.306062 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.274920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.27.1.253:37827)
0807 18:28:32.275174 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.275183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.287786 (+ 12603us) server_negotiation.cc:407] Connection header received
0807 18:28:32.287994 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.288000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.288086 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.288158 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:32.289784 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.290858 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.293297 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.294059 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.294162 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.298405 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:32.298422 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:32.298428 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:32.298475 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:32.302519 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.303401 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.303407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.303411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.303567 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.304084 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.304092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.304096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.304465 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:32.304600 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.305410 (+ 810us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.305655 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 18:28:32.306139 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.274606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:37827 (local address 127.0.0.1:39402)
0807 18:28:32.287681 (+ 13075us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:32.287698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:32.287819 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:32.288381 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:32.288387 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:32.289128 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:32.289597 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.289612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.291007 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.291011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.293110 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.293126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.294513 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.294528 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.294729 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.295462 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:32.295488 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:32.298102 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:32.302710 (+ 4608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.302715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.302730 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.303258 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.303720 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.303725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.303729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.303937 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.304689 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:32.304694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:32.305088 (+ 394us) client_negotiation.cc:769] Sending connection context
0807 18:28:32.305252 (+ 164us) client_negotiation.cc:240] Negotiation successful
0807 18:28:32.305540 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12926,"mutex_wait_us":594,"thread_start_us":11824,"threads_started":1}
I20260807 18:28:32.309175 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 1802
I20260807 18:28:32.311162 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.275179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46318)
0807 18:28:32.286416 (+ 11237us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:32.286436 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:32.286644 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:32.287168 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:32.287175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:32.288338 (+ 1163us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:32.288780 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.288793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.290399 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.290404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.292533 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.292547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.293853 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.293862 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.294025 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.294651 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:32.294673 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:32.296033 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:32.300107 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.300117 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.300121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.300688 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.301106 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.301111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.301114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.301315 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.302004 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:32.302008 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:32.302282 (+ 274us) client_negotiation.cc:769] Sending connection context
0807 18:28:32.303870 (+ 1588us) client_negotiation.cc:240] Negotiation successful
0807 18:28:32.304215 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10204,"mutex_wait_us":49,"spinlock_wait_cycles":7552,"thread_start_us":9025,"threads_started":1}
Aug 07 18:28:32 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.193@KRBTEST.COM
Aug 07 18:28:32 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127309, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.254@KRBTEST.COM for oryx/127.27.1.252@KRBTEST.COM
I20260807 18:28:32.332979 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.310413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.27.1.193:35689)
0807 18:28:32.311932 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.311943 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.311973 (+ 30us) server_negotiation.cc:407] Connection header received
0807 18:28:32.312049 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.312059 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.312386 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.312633 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:32.314463 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.316077 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.318028 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.318920 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.319460 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.324099 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:32.324110 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:32.324124 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:32.324164 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:32.328428 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.329372 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.329383 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.329388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.329545 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.330133 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.330140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.330144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.330600 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:32.330897 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.332216 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.332498 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":1231,"threads_started":1}
I20260807 18:28:32.334649 1933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:32.338176 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.318386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33443 (local address 127.0.0.1:51332)
0807 18:28:32.318601 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:32.318620 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:32.318724 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:32.319243 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:32.319249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:32.319756 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:32.320102 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.320115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.321759 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.321765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.323287 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.323304 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.324376 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.324383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.324550 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.325517 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:32.325543 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:32.328541 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:32.332815 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.332821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.332824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.334530 (+ 1706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.335088 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.335097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.335103 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.335331 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.336464 (+ 1133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:32.336473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:32.336828 (+ 355us) client_negotiation.cc:769] Sending connection context
0807 18:28:32.337010 (+ 182us) client_negotiation.cc:240] Negotiation successful
0807 18:28:32.337534 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 18:28:32.339062 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.318501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.27.1.252:33443)
0807 18:28:32.318656 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.318660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.318675 (+ 15us) server_negotiation.cc:407] Connection header received
0807 18:28:32.318904 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.318909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.319025 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.320263 (+ 1238us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:32.320455 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.321602 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.323459 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.324206 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.324341 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.328716 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:32.328727 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:32.328730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:32.328759 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:32.332676 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.334685 (+ 2009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.334691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.334694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.334844 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.335613 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.335646 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.335652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.336272 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:32.338576 (+ 2304us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.338670 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.338833 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:28:32.340497 1732 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.1.252:33443: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d"
member_type: NON_VOTER
last_known_addr {
host: "127.27.1.252"
port: 33443
}
attrs {
promote: true
}
}
dest_uuid: "c98333801763413d99a0d26660b81d1a"
cas_config_opid_index: 5
I20260807 18:28:32.341387 1732 consensus_queue.cc:237] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } }
I20260807 18:28:32.343168 2337 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9bb886b775c34a0d964e7dd52eb2518d (127.27.1.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.348155 2393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 LEARNER]: Refusing update from remote peer c98333801763413d99a0d26660b81d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 18:28:32.350226 2483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:32.356400 2337 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.1.252:33443
I20260807 18:28:32.357147 2341 master.cc:537] Master@127.27.1.252:33443 shutting down...
I20260807 18:28:32.358676 2483 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Refusing update from remote peer c98333801763413d99a0d26660b81d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 18:28:32.360062 2544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 18:28:32.374701 2541 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:32.378158 2544 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.378633 2544 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:32.379918 2393 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9bb886b775c34a0d964e7dd52eb2518d (127.27.1.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.382860 2518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:32.383208 2518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } }
I20260807 18:28:32.384956 2423 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: NON_VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: true } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.385267 2423 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: This master's current role is: LEARNER
I20260807 18:28:32.385619 2544 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a: attempting to promote NON_VOTER eec35020298c422d8dfbbea46d5dd5cc to VOTER
I20260807 18:28:32.386371 2544 consensus_queue.cc:237] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } }
I20260807 18:28:32.388841 2393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 LEARNER]: Refusing update from remote peer c98333801763413d99a0d26660b81d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 18:28:32.389458 2544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 18:28:32.392867 2518 leader_election.cc:288] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c98333801763413d99a0d26660b81d1a (127.27.1.254:38033), eec35020298c422d8dfbbea46d5dd5cc (127.27.1.253:37827)
I20260807 18:28:32.394196 2392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bb886b775c34a0d964e7dd52eb2518d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eec35020298c422d8dfbbea46d5dd5cc" is_pre_election: true
I20260807 18:28:32.397091 2341 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:32.397578 2341 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:32.397784 2341 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: stopping tablet replica
W20260807 18:28:32.402397 2453 proxy.cc:239] Call had error, refreshing address and retrying: Service unavailable: reactor is shutting down (error 108)
I20260807 18:28:32.403162 1744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bb886b775c34a0d964e7dd52eb2518d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c98333801763413d99a0d26660b81d1a" is_pre_election: true
W20260807 18:28:32.404385 1718 consensus_peers.cc:487] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a -> Peer 9bb886b775c34a0d964e7dd52eb2518d (127.27.1.252:33443): Couldn't send request to peer 9bb886b775c34a0d964e7dd52eb2518d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 18:28:32.409544 2542 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, eec35020298c422d8dfbbea46d5dd5cc (127.27.1.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.410563 2393 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, eec35020298c422d8dfbbea46d5dd5cc (127.27.1.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.415218 2541 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.415621 2541 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: This master's current role is: LEADER
W20260807 18:28:32.415884 2548 leader_election.cc:334] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c98333801763413d99a0d26660b81d1a (127.27.1.254:38033): Service unavailable: reactor is shutting down (error 108)
I20260807 18:28:32.416002 2541 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.416097 2548 leader_election.cc:302] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bb886b775c34a0d964e7dd52eb2518d; no voters: c98333801763413d99a0d26660b81d1a, eec35020298c422d8dfbbea46d5dd5cc
I20260807 18:28:32.416375 2541 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: This master's current role is: LEADER
W20260807 18:28:32.416498 2548 raft_consensus.cc:2670] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260807 18:28:32.416785 1718 connection.cc:535] server connection from 127.0.0.1:46306 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 18:28:32.417591 2423 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.417904 2423 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:28:32.421164 2341 master.cc:552] Master@127.27.1.252:33443 shutdown complete.
I20260807 18:28:32.421298 2341 master_runner.cc:305] Clearing existing system tablet
I20260807 18:28:32.421694 2341 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:32.421835 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:32.424644 2341 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 18:28:32.432174 2550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:32.432569 2341 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:32.432883 2341 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.007s
I20260807 18:28:32.432996 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal
uuid: "9bb886b775c34a0d964e7dd52eb2518d"
format_stamp: "Formatted at 2026-08-07 18:28:32 on dist-test-slave-1xrh"
server_key: "becc20323fbfb7cbe06a32bbd4bf1df0"
server_key_iv: "34be01b0a7e7acbaf51022fd983e4cf9"
server_key_version: "encryptionkey@0"
I20260807 18:28:32.436843 2341 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:32.441882 2341 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 18:28:32.442070 2341 log.cc:1196] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 18:28:32.442585 2341 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Deleting consensus metadata
I20260807 18:28:32.442879 2341 master_runner.cc:315] Copying system tablet from 127.27.1.254:38033
I20260807 18:28:32.445730 2341 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.1.254:38033
I20260807 18:28:32.459796 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.446547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.27.1.254:38033)
0807 18:28:32.446818 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.446827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.447754 (+ 927us) server_negotiation.cc:407] Connection header received
0807 18:28:32.447992 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.447997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.448078 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.448191 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:32.449067 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.449888 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.451200 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.451620 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.451763 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.453833 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:32.453838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:32.453841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:32.453877 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:32.456957 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.457607 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.457614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.457617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.457765 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:32.458184 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:32.458189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:32.458193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:32.458484 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:32.458650 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.459159 (+ 509us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.459402 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260807 18:28:32.459903 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.446532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46332)
0807 18:28:32.447626 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:32.447644 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:32.447823 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:32.448225 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:32.448229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:32.448663 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:32.448911 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.448919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.450046 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.450052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.451016 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.451028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.451780 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.451786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.451988 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.452796 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:32.452815 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:32.453617 (+ 802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:32.457116 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.457121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.457125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.457487 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.457891 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:32.457896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:32.457900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:32.458071 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:32.458612 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:32.458617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:32.458798 (+ 181us) client_negotiation.cc:769] Sending connection context
0807 18:28:32.459024 (+ 226us) client_negotiation.cc:240] Negotiation successful
0807 18:28:32.459388 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":767,"threads_started":1}
I20260807 18:28:32.460776 1757 tablet_copy_service.cc:140] P c98333801763413d99a0d26660b81d1a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9bb886b775c34a0d964e7dd52eb2518d ({username='oryx', principal='oryx/127.27.1.252@KRBTEST.COM'} at 127.0.0.1:46332)
I20260807 18:28:32.460959 1757 tablet_copy_service.cc:161] P c98333801763413d99a0d26660b81d1a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9bb886b775c34a0d964e7dd52eb2518d at {username='oryx', principal='oryx/127.27.1.252@KRBTEST.COM'} at 127.0.0.1:46332: session id = 9bb886b775c34a0d964e7dd52eb2518d-00000000000000000000000000000000
I20260807 18:28:32.463549 1757 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 18:28:32.465217 2341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 18:28:32.470165 2341 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: tablet copy: Starting download of 0 data blocks...
I20260807 18:28:32.470525 2341 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: tablet copy: Starting download of 1 WAL segments...
I20260807 18:28:32.472958 2341 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 18:28:32.476115 1757 tablet_copy_service.cc:342] P c98333801763413d99a0d26660b81d1a: Request end of tablet copy session 9bb886b775c34a0d964e7dd52eb2518d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.1.252@KRBTEST.COM'} at 127.0.0.1:46332
I20260807 18:28:32.476256 1757 tablet_copy_service.cc:433] P c98333801763413d99a0d26660b81d1a: ending tablet copy session 9bb886b775c34a0d964e7dd52eb2518d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9bb886b775c34a0d964e7dd52eb2518d
I20260807 18:28:32.487262 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:32.490075 2558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:32.496161 2559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:32.499584 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 1802
W20260807 18:28:32.506340 2561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:32.507216 2341 server_base.cc:1034] running on GCE node
I20260807 18:28:32.507588 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:32.508327 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:32.509539 2341 hybrid_clock.cc:648] HybridClock initialized: now 1786127312509485 us; error 42 us; skew 500 ppm
I20260807 18:28:32.509837 2341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:32 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127312, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:32.512193 2341 init.cc:377] Logged in from keytab as oryx/127.27.1.252@KRBTEST.COM (short username oryx)
I20260807 18:28:32.523663 2341 webserver.cc:460] Webserver started at http://127.27.1.252:46179/ using document root <none> and password file <none>
I20260807 18:28:32.524056 2341 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:32.524170 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:32.527731 2341 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 18:28:32.529886 2568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:32.530391 2341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 18:28:32.530567 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:35689
--local_ip_for_outbound_sockets=127.27.1.193
--tserver_master_addrs=127.27.1.254:38033
--webserver_port=44231
--webserver_interface=127.27.1.193
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab
--principal=oryx/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.1.254:38033,127.27.1.253:37827
--tserver_master_addrs=127.27.1.254:38033,127.27.1.253:37827,127.27.1.252:33443 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:28:32.530526 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal
uuid: "9bb886b775c34a0d964e7dd52eb2518d"
format_stamp: "Formatted at 2026-08-07 18:28:32 on dist-test-slave-1xrh"
server_key: "becc20323fbfb7cbe06a32bbd4bf1df0"
server_key_iv: "34be01b0a7e7acbaf51022fd983e4cf9"
server_key_version: "encryptionkey@0"
I20260807 18:28:32.530723 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:32.553598 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:32.558393 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:32.583899 2341 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.252:33443
I20260807 18:28:32.583951 2622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.252:33443 every 8 connection(s)
I20260807 18:28:32.584964 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
I20260807 18:28:32.590472 2624 sys_catalog.cc:263] Verifying existing consensus state
I20260807 18:28:32.594630 2624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Bootstrap starting.
I20260807 18:28:32.614454 2624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 18:28:32.615000 2624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Bootstrap complete.
I20260807 18:28:32.616225 2624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } }
I20260807 18:28:32.616544 2624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9bb886b775c34a0d964e7dd52eb2518d, State: Initialized, Role: LEARNER
I20260807 18:28:32.616803 2624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } }
I20260807 18:28:32.617677 2629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.617983 2624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:32.618144 2629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: This master's current role is: LEARNER
I20260807 18:28:32.631449 2641 catalog_manager.cc:1259] Loaded cluster ID: 1bcd0fbcc44d441687caedeef987db55
I20260807 18:28:32.631659 2641 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: loading cluster ID for follower catalog manager: success
I20260807 18:28:32.635459 2641 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: acquiring CA information for follower catalog manager: success
I20260807 18:28:32.637952 2641 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 18:28:32.908417 2572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:32.908731 2572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:32.908913 2572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:32.916901 2572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:32.917286 2572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:32.917510 2572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:32.927963 2572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab
--principal=oryx/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:35689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=44231
--tserver_master_addrs=127.27.1.254:38033,127.27.1.253:37827,127.27.1.252:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:32.929375 2572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:32.930848 2572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:32.944749 2650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:32.944885 2647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:32.944818 2648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:32.945802 2572 server_base.cc:1034] running on GCE node
I20260807 18:28:32.946412 2572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:32.947387 2572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:32.948612 2572 hybrid_clock.cc:648] HybridClock initialized: now 1786127312948558 us; error 41 us; skew 500 ppm
I20260807 18:28:32.948920 2572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:32 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127312, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:32.952097 2572 init.cc:377] Logged in from keytab as oryx/127.27.1.193@KRBTEST.COM (short username oryx)
I20260807 18:28:32.953583 2572 webserver.cc:460] Webserver started at http://127.27.1.193:44231/ using document root <none> and password file <none>
I20260807 18:28:32.954276 2572 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:32.954455 2572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:32.959684 2572 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:28:32.962366 2658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:32.963732 2572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:32.963932 2572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "ebf04659d5b944b2ab47b4a7d0f8e918"
format_stamp: "Formatted at 2026-08-07 18:28:29 on dist-test-slave-1xrh"
server_key: "fa4ae1039df1c1b875648e586a289b35"
server_key_iv: "78bf4212ec73ab8903687137b3f6f2cd"
server_key_version: "encryptionkey@0"
I20260807 18:28:32.964958 2572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:32.983837 2572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:32.988933 2572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:32.989219 2572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:32.990468 2572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:32.992086 2572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:32.992173 2572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:32.992331 2572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:32.992450 2572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:32.993754 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.986661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.27.1.252:33443)
0807 18:28:32.987416 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:32.987423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:32.987446 (+ 23us) server_negotiation.cc:407] Connection header received
0807 18:28:32.987531 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:32.987535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:32.987606 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:32.987711 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:32.988219 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.989313 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.991619 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.992710 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.992868 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.993068 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:32.993280 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 18:28:32.993445 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":463,"threads_started":1}
I20260807 18:28:32.993813 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:32.986528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33443 (local address 127.0.0.1:51334)
0807 18:28:32.987031 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:32.987052 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:32.987231 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:32.987778 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:32.987785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:32.987818 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:32.988065 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.988078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.989489 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.989494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.991427 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:32.991445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:32.992904 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:32.992911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:32.993093 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:32.993137 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 18:28:32.993251 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 18:28:32.993412 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":225,"threads_started":1}
I20260807 18:28:32.994813 2596 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260807 18:28:32.995406 2629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c98333801763413d99a0d26660b81d1a. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:32.995721 2629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: This master's current role is: LEARNER
I20260807 18:28:33.031260 2572 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:35689
I20260807 18:28:33.031349 2781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:35689 every 8 connection(s)
I20260807 18:28:33.032873 2572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:33.041152 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 2572
I20260807 18:28:33.041554 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 1944
Aug 07 18:28:33 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127312, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.252@KRBTEST.COM
Aug 07 18:28:33 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127312, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:33.067047 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.038180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:33999 (local address 127.27.1.252:33443)
0807 18:28:33.038440 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:33.038450 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:33.039594 (+ 1144us) server_negotiation.cc:407] Connection header received
0807 18:28:33.045336 (+ 5742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:33.045346 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:33.045470 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:33.045560 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:33.048300 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.049648 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.052731 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.053528 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.053643 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.059525 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:33.059531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:33.059534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:33.059573 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:33.063262 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.064246 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.064251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.064255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.064426 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.064946 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.064952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.064955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.065321 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:33.065463 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:33.066422 (+ 959us) server_negotiation.cc:299] Negotiation successful
0807 18:28:33.066666 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260807 18:28:33.069585 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.036729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33443 (local address 127.27.1.193:33999)
0807 18:28:33.039435 (+ 2706us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:33.039480 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:33.042833 (+ 3353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:33.046241 (+ 3408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:33.046255 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:33.047164 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:33.048066 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.048095 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.049903 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.049912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.052542 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.052561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.054057 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.054066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.054483 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.055398 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:33.055434 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:33.059218 (+ 3784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:33.063434 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.063444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.063459 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.064107 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.064572 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.064576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.064580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.064813 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.065557 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:33.065566 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:33.065968 (+ 402us) client_negotiation.cc:769] Sending connection context
0807 18:28:33.067354 (+ 1386us) client_negotiation.cc:240] Negotiation successful
0807 18:28:33.068004 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2405,"mutex_wait_us":3,"thread_start_us":29326,"threads_started":1}
Aug 07 18:28:33 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127312, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.253@KRBTEST.COM
Aug 07 18:28:33 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.194@KRBTEST.COM
I20260807 18:28:33.071295 2782 heartbeater.cc:347] Connected to a master server at 127.27.1.252:33443
I20260807 18:28:33.071571 2782 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:33.071703 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.036069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:47351 (local address 127.27.1.254:38033)
0807 18:28:33.049496 (+ 13427us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:33.049505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:33.050456 (+ 951us) server_negotiation.cc:407] Connection header received
0807 18:28:33.050541 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:33.050549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:33.050651 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:33.050772 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:33.052658 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.054111 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.056835 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.057426 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.057531 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.062717 (+ 5186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:33.062725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:33.062732 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:33.062786 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:33.066755 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.068150 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.068157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.068161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.068326 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.068833 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.068841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.068845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.069246 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:33.069421 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:33.070914 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0807 18:28:33.071177 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13245,"thread_start_us":14228,"threads_started":1}
I20260807 18:28:33.072533 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.037037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.193:47351)
0807 18:28:33.050014 (+ 12977us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:33.050040 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:33.050203 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:33.051074 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:33.051081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:33.052002 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:33.052432 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.052448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.054323 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.054331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.056654 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.056671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.057801 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.057807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.058145 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.058873 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:33.058905 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:33.062508 (+ 3603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:33.066937 (+ 4429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.066941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.066946 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.067993 (+ 1047us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.068474 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.068480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.068485 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.068688 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.069492 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:33.069497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:33.069948 (+ 451us) client_negotiation.cc:769] Sending connection context
0807 18:28:33.070129 (+ 181us) client_negotiation.cc:240] Negotiation successful
0807 18:28:33.070429 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12822,"mutex_wait_us":103,"thread_start_us":12625,"threads_started":1}
I20260807 18:28:33.072563 2782 heartbeater.cc:511] Master 127.27.1.252:33443 requested a full tablet report, sending...
I20260807 18:28:33.074213 2783 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38033
I20260807 18:28:33.074352 2783 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:33.074820 2783 heartbeater.cc:511] Master 127.27.1.254:38033 requested a full tablet report, sending...
I20260807 18:28:33.076437 2586 ts_manager.cc:194] Registered new tserver with Master: ebf04659d5b944b2ab47b4a7d0f8e918 (127.27.1.193:35689)
I20260807 18:28:33.077565 1732 ts_manager.cc:194] Re-registered known tserver with Master: ebf04659d5b944b2ab47b4a7d0f8e918 (127.27.1.193:35689)
I20260807 18:28:33.078254 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.036729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:37827 (local address 127.27.1.193:48779)
0807 18:28:33.060172 (+ 23443us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:33.060193 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:33.060326 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:33.061001 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:33.061006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:33.061556 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:33.061949 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.061968 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.063516 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.063523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.065138 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.065151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.066563 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.066572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.066742 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.067351 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:33.067373 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:33.070823 (+ 3450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:33.075426 (+ 4603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.075431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.075434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.075904 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.076337 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.076342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.076346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.076526 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.077287 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:33.077294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:33.077446 (+ 152us) client_negotiation.cc:769] Sending connection context
0807 18:28:33.077594 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 18:28:33.077869 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23279,"mutex_wait_us":1708,"thread_start_us":488,"threads_started":1}
I20260807 18:28:33.078660 1732 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:47351
I20260807 18:28:33.079229 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.037578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:48779 (local address 127.27.1.253:37827)
0807 18:28:33.059622 (+ 22044us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:33.059629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:33.060439 (+ 810us) server_negotiation.cc:407] Connection header received
0807 18:28:33.060530 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:33.060535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:33.060635 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:33.060786 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:33.062174 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.063360 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.065301 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.065977 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.066106 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.071042 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:33.071048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:33.071053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:33.071101 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:33.075230 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.076049 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.076054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.076058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.076200 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.076664 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.076669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.076673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.077082 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:33.077223 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:33.078583 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0807 18:28:33.078802 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21909,"thread_start_us":22053,"threads_started":1}
I20260807 18:28:33.079349 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.042980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.27.1.194:39689)
0807 18:28:33.056920 (+ 13940us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:33.056930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:33.056954 (+ 24us) server_negotiation.cc:407] Connection header received
0807 18:28:33.057019 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:33.057026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:33.057255 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:33.060039 (+ 2784us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:33.060428 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.062489 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.064728 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.065661 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.071917 (+ 6256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.072218 (+ 301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:33.072230 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:33.072245 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:33.072289 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:33.076143 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.076912 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.076925 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.076927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.077061 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.077450 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.077457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.077465 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.077832 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:33.078134 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:33.078436 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 18:28:33.078686 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13745,"thread_start_us":14383,"threads_started":1}
I20260807 18:28:33.080051 2784 heartbeater.cc:347] Connected to a master server at 127.27.1.253:37827
I20260807 18:28:33.080201 2784 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:33.080741 2077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:33.081167 2784 heartbeater.cc:511] Master 127.27.1.253:37827 requested a full tablet report, sending...
I20260807 18:28:33.082938 2383 ts_manager.cc:194] Registered new tserver with Master: ebf04659d5b944b2ab47b4a7d0f8e918 (127.27.1.193:35689)
I20260807 18:28:33.163899 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 1944
I20260807 18:28:33.183799 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:39689
--local_ip_for_outbound_sockets=127.27.1.194
--tserver_master_addrs=127.27.1.254:38033
--webserver_port=33173
--webserver_interface=127.27.1.194
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab
--principal=oryx/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.1.254:38033,127.27.1.253:37827
--tserver_master_addrs=127.27.1.254:38033,127.27.1.253:37827,127.27.1.252:33443 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:28:33.386456 2544 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: NON_VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: true } } }
I20260807 18:28:33.386894 2544 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:33.388600 2544 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a: attempting to promote NON_VOTER 9bb886b775c34a0d964e7dd52eb2518d to VOTER
I20260807 18:28:33.389237 2544 consensus_queue.cc:237] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } }
I20260807 18:28:33.390990 2596 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 LEARNER]: Refusing update from remote peer c98333801763413d99a0d26660b81d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 18:28:33.391072 2393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Refusing update from remote peer c98333801763413d99a0d26660b81d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 18:28:33.391597 2544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 18:28:33.391989 2796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 18:28:33.394567 2544 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9bb886b775c34a0d964e7dd52eb2518d (127.27.1.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:33.395519 2393 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9bb886b775c34a0d964e7dd52eb2518d (127.27.1.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:33.398602 2544 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:33.398869 2544 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:33.399259 2544 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:33.399559 2544 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:33.401108 2800 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:33.401510 2800 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:28:33.401786 2796 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:33.402076 2796 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:33.402335 2596 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9bb886b775c34a0d964e7dd52eb2518d (127.27.1.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:33.405308 2629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c98333801763413d99a0d26660b81d1a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:33.405560 2629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: This master's current role is: FOLLOWER
W20260807 18:28:33.529443 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:33.529762 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:33.529933 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:33.537421 2794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:33.537565 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:33.537734 2794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:33.547752 2794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab
--principal=oryx/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:39689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=33173
--tserver_master_addrs=127.27.1.254:38033,127.27.1.253:37827,127.27.1.252:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:33.549124 2794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:33.550475 2794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:33.563652 2807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:33.563838 2806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:33.563683 2809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:33.565519 2794 server_base.cc:1034] running on GCE node
I20260807 18:28:33.566102 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:33.566992 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:33.568187 2794 hybrid_clock.cc:648] HybridClock initialized: now 1786127313568130 us; error 56 us; skew 500 ppm
I20260807 18:28:33.568545 2794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:33 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127313, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:33.571863 2794 init.cc:377] Logged in from keytab as oryx/127.27.1.194@KRBTEST.COM (short username oryx)
I20260807 18:28:33.573436 2794 webserver.cc:460] Webserver started at http://127.27.1.194:33173/ using document root <none> and password file <none>
I20260807 18:28:33.574115 2794 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:33.574275 2794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:33.579659 2794 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:28:33.582495 2816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:33.583896 2794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:33.584074 2794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "698923a8cfda4e21a16dbf1b1239f5bf"
format_stamp: "Formatted at 2026-08-07 18:28:30 on dist-test-slave-1xrh"
server_key: "14103f01284fe70d7afa5b26b39aff78"
server_key_iv: "7a7636968a11168afbac123da1437981"
server_key_version: "encryptionkey@0"
I20260807 18:28:33.585004 2794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:33.623087 2794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:33.627444 2794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:33.627683 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:33.628561 2794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:33.629850 2794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:33.629933 2794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:33.630065 2794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:33.630149 2794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:33.663054 2794 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:39689
I20260807 18:28:33.663153 2938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:39689 every 8 connection(s)
I20260807 18:28:33.664577 2794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:33.669075 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 2794
I20260807 18:28:33.669423 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 2089
Aug 07 18:28:33 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127313, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.252@KRBTEST.COM
I20260807 18:28:33.722695 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.668281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:44007 (local address 127.27.1.252:33443)
0807 18:28:33.682929 (+ 14648us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:33.682936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:33.682962 (+ 26us) server_negotiation.cc:407] Connection header received
0807 18:28:33.688239 (+ 5277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:33.688249 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:33.688371 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:33.688478 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:33.695899 (+ 7421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.697609 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.704161 (+ 6552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.705124 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.705250 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.714980 (+ 9730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:33.714998 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:33.715003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:33.715043 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:33.718833 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.719773 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.719778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.719782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.719936 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.720512 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.720517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.720521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.720864 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:33.721012 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:33.722009 (+ 997us) server_negotiation.cc:299] Negotiation successful
0807 18:28:33.722274 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14509,"thread_start_us":14393,"threads_started":1}
Aug 07 18:28:33 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.195@KRBTEST.COM
Aug 07 18:28:33 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127313, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:33.725977 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.668470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33443 (local address 127.27.1.194:44007)
0807 18:28:33.680300 (+ 11830us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:33.680378 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:33.683676 (+ 3298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:33.689488 (+ 5812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:33.690669 (+ 1181us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:33.691723 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:33.692806 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.692837 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.697931 (+ 5094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.697939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.700803 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.700825 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.705771 (+ 4946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.705781 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.706241 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.708106 (+ 1865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:33.708147 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:33.714428 (+ 6281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:33.719023 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.719031 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.719047 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.719631 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.720110 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.720115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.720119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.720364 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.721111 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:33.721123 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:33.721548 (+ 425us) client_negotiation.cc:769] Sending connection context
0807 18:28:33.723022 (+ 1474us) client_negotiation.cc:240] Negotiation successful
0807 18:28:33.724344 (+ 1322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11426,"thread_start_us":47553,"threads_started":1}
Aug 07 18:28:33 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127313, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.253@KRBTEST.COM
I20260807 18:28:33.729015 2939 heartbeater.cc:347] Connected to a master server at 127.27.1.252:33443
I20260807 18:28:33.729274 2939 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:33.730198 2939 heartbeater.cc:511] Master 127.27.1.252:33443 requested a full tablet report, sending...
I20260807 18:28:33.732445 2586 ts_manager.cc:194] Registered new tserver with Master: 698923a8cfda4e21a16dbf1b1239f5bf (127.27.1.194:39689)
I20260807 18:28:33.733297 2949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.668798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.194:48981)
0807 18:28:33.713702 (+ 44904us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:33.713728 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:33.713886 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:33.716850 (+ 2964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:33.716857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:33.717763 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:33.718182 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.718198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.719773 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.719778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.721345 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.721358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.722541 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.722549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.722725 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.723321 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:33.723342 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:33.726046 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:33.730101 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.730108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.730112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.730856 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.731380 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.731385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.731390 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.731550 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.732247 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:33.732251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:33.732401 (+ 150us) client_negotiation.cc:769] Sending connection context
0807 18:28:33.732534 (+ 133us) client_negotiation.cc:240] Negotiation successful
0807 18:28:33.732859 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44707,"thread_start_us":44471,"threads_started":1}
I20260807 18:28:33.734388 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.667737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:48981 (local address 127.27.1.254:38033)
0807 18:28:33.715000 (+ 47263us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:33.715009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:33.715034 (+ 25us) server_negotiation.cc:407] Connection header received
0807 18:28:33.715112 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:33.715141 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:33.715247 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:33.718342 (+ 3095us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:33.718521 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.719624 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.721545 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.722334 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.723050 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.726298 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:33.726305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:33.726309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:33.726352 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:33.729926 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.731038 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.731042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.731046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.731239 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.731708 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.731713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.731715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.732110 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:33.733635 (+ 1525us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:33.733768 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 18:28:33.734018 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47088,"thread_start_us":46957,"threads_started":1}
I20260807 18:28:33.734581 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.668470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:37827 (local address 127.27.1.194:55887)
0807 18:28:33.716462 (+ 47992us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:33.716481 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:33.716615 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:33.717520 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:33.717525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:33.718298 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:33.718633 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.718646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.720207 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.720213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.722031 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:33.722051 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.723398 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.723409 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:33.723632 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.724274 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:33.724305 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:33.727416 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:33.731609 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.731614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.731618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.732114 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.732552 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:33.732574 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:33.732578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:33.732739 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:33.733480 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:33.733498 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:33.733641 (+ 143us) client_negotiation.cc:769] Sending connection context
0807 18:28:33.733852 (+ 211us) client_negotiation.cc:240] Negotiation successful
0807 18:28:33.734178 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47834,"mutex_wait_us":582,"thread_start_us":9212,"threads_started":1}
I20260807 18:28:33.734987 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.667993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:55887 (local address 127.27.1.253:37827)
0807 18:28:33.714463 (+ 46470us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:33.714472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:33.716609 (+ 2137us) server_negotiation.cc:407] Connection header received
0807 18:28:33.716788 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:33.716793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:33.717173 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:33.717292 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:33.718808 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.720007 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.722213 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.722932 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.723054 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.727657 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:33.727666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:33.727671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:33.727736 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:33.731456 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.732246 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.732251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.732255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.732400 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.732882 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.732887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.732890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.733235 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:33.733375 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:33.733994 (+ 619us) server_negotiation.cc:299] Negotiation successful
0807 18:28:33.734224 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46297,"thread_start_us":46481,"threads_started":1}
I20260807 18:28:33.735177 2940 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38033
I20260807 18:28:33.735320 2940 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:33.735827 2940 heartbeater.cc:511] Master 127.27.1.254:38033 requested a full tablet report, sending...
I20260807 18:28:33.736011 2942 heartbeater.cc:347] Connected to a master server at 127.27.1.253:37827
I20260807 18:28:33.736263 2942 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:33.736277 2952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:33.671069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.27.1.195:34445)
0807 18:28:33.707832 (+ 36763us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:33.707845 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:33.709268 (+ 1423us) server_negotiation.cc:407] Connection header received
0807 18:28:33.709366 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:33.709375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:33.709709 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:33.709908 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:33.713721 (+ 3813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.715330 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.717523 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:33.718496 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:33.723708 (+ 5212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:33.727095 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:33.727110 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:33.727149 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:33.727204 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:33.731802 (+ 4598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.732712 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.732721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.732724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.732933 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:33.733402 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:33.733407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:33.733410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:33.733920 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:33.734222 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:33.735518 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0807 18:28:33.735790 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36513,"thread_start_us":36772,"threads_started":1}
I20260807 18:28:33.736727 2942 heartbeater.cc:511] Master 127.27.1.253:37827 requested a full tablet report, sending...
I20260807 18:28:33.737125 1732 ts_manager.cc:194] Re-registered known tserver with Master: 698923a8cfda4e21a16dbf1b1239f5bf (127.27.1.194:39689)
I20260807 18:28:33.737640 2220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:33.737916 2383 ts_manager.cc:194] Registered new tserver with Master: 698923a8cfda4e21a16dbf1b1239f5bf (127.27.1.194:39689)
I20260807 18:28:33.738550 1732 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:48981
I20260807 18:28:33.822165 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 2089
I20260807 18:28:33.841979 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:34445
--local_ip_for_outbound_sockets=127.27.1.195
--tserver_master_addrs=127.27.1.254:38033
--webserver_port=44021
--webserver_interface=127.27.1.195
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab
--principal=oryx/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.1.254:38033,127.27.1.253:37827
--tserver_master_addrs=127.27.1.254:38033,127.27.1.253:37827,127.27.1.252:33443 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:28:34.082052 2783 heartbeater.cc:503] Master 127.27.1.254:38033 was elected leader, sending a full tablet report...
W20260807 18:28:34.180603 2954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:34.180879 2954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:34.181039 2954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:34.188337 2954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:34.188462 2954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:34.188685 2954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:34.198872 2954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab
--principal=oryx/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:34445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=44021
--tserver_master_addrs=127.27.1.254:38033,127.27.1.253:37827,127.27.1.252:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:34.200246 2954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:34.201584 2954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:34.215615 2964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:34.215668 2962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:34.215698 2961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:34.216529 2954 server_base.cc:1034] running on GCE node
I20260807 18:28:34.217149 2954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:34.218400 2954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:34.219637 2954 hybrid_clock.cc:648] HybridClock initialized: now 1786127314219594 us; error 52 us; skew 500 ppm
I20260807 18:28:34.219964 2954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:34 dist-test-slave-1xrh krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127314, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:34.228837 2954 init.cc:377] Logged in from keytab as oryx/127.27.1.195@KRBTEST.COM (short username oryx)
I20260807 18:28:34.230345 2954 webserver.cc:460] Webserver started at http://127.27.1.195:44021/ using document root <none> and password file <none>
I20260807 18:28:34.231074 2954 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:34.231266 2954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:34.237295 2954 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260807 18:28:34.239989 2972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:34.241284 2954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:34.241463 2954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "3eb0a9e28caf420487cc3e2db20f96c3"
format_stamp: "Formatted at 2026-08-07 18:28:30 on dist-test-slave-1xrh"
server_key: "495f7a271843426048e04bc054e07641"
server_key_iv: "3fca96b9856aff0a569e35f285c1cd2b"
server_key_version: "encryptionkey@0"
I20260807 18:28:34.242424 2954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:34.275213 2954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:34.279218 2954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:34.279515 2954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:34.280489 2954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:34.281807 2954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:34.281929 2954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:34.282078 2954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:34.282169 2954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:34.316567 2954 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:34445
I20260807 18:28:34.316692 3100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:34445 every 8 connection(s)
I20260807 18:28:34.318079 2954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:34.326535 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 2954
Aug 07 18:28:34 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127314, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.252@KRBTEST.COM
Aug 07 18:28:34 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127314, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:34.356137 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.322311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.195:36823)
0807 18:28:34.327777 (+ 5466us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:34.327831 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:34.330919 (+ 3088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:34.332950 (+ 2031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:34.332958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:34.333592 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:34.334379 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.334402 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.336016 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.336020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.338194 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.338209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.339522 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.339530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.339807 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.340673 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:34.340706 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:34.344957 (+ 4251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:34.350494 (+ 5537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:34.350508 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:34.350532 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:34.351228 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:34.351785 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:34.351793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:34.351799 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:34.352101 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:34.352939 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:34.352950 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:34.353459 (+ 509us) client_negotiation.cc:769] Sending connection context
0807 18:28:34.353800 (+ 341us) client_negotiation.cc:240] Negotiation successful
0807 18:28:34.354461 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5164,"thread_start_us":10451,"threads_started":1}
I20260807 18:28:34.357525 3107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.321761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:36823 (local address 127.27.1.254:38033)
0807 18:28:34.331790 (+ 10029us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.331799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.331826 (+ 27us) server_negotiation.cc:407] Connection header received
0807 18:28:34.331908 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.331916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.332027 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.332170 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:34.334633 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.335784 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.338370 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.339046 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.339195 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.345256 (+ 6061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:34.345267 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:34.345274 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:34.345332 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:34.350289 (+ 4957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.351429 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.351437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.351443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.351626 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.352253 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.352261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.352266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.352684 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:34.352836 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.356713 (+ 3877us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.357025 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9856,"thread_start_us":9747,"threads_started":1}
I20260807 18:28:34.359781 3102 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38033
I20260807 18:28:34.360325 3102 heartbeater.cc:464] Registering TS with master...
Aug 07 18:28:34 dist-test-slave-1xrh krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127314, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.253@KRBTEST.COM
I20260807 18:28:34.362973 3102 heartbeater.cc:511] Master 127.27.1.254:38033 requested a full tablet report, sending...
I20260807 18:28:34.367472 1732 ts_manager.cc:194] Re-registered known tserver with Master: 3eb0a9e28caf420487cc3e2db20f96c3 (127.27.1.195:34445)
I20260807 18:28:34.367558 3106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.321577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:32865 (local address 127.27.1.252:33443)
0807 18:28:34.332676 (+ 11099us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.332681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.334313 (+ 1632us) server_negotiation.cc:407] Connection header received
0807 18:28:34.334519 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.334526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.334679 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.334798 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:34.336132 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.337277 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.339250 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.339867 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.339963 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.358725 (+ 18762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:34.358735 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:34.358743 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:34.358800 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:34.363407 (+ 4607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.364504 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.364514 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.364520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.364708 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.365293 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.365301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.365307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.365754 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:34.365921 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.366662 (+ 741us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.366978 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10984,"thread_start_us":10881,"threads_started":1}
I20260807 18:28:34.368777 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.322311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33443 (local address 127.27.1.195:32865)
0807 18:28:34.334156 (+ 11845us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:34.334180 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:34.334359 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:34.335049 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:34.335054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:34.335641 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:34.335942 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.335954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.337466 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.337475 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.339068 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.339083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.340233 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.340238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.340364 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.341024 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:34.341045 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:34.358443 (+ 17398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:34.363612 (+ 5169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:34.363616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:34.363620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:34.364323 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:34.364891 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:34.364899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:34.364905 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:34.365136 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:34.366043 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:34.366051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:34.366268 (+ 217us) client_negotiation.cc:769] Sending connection context
0807 18:28:34.367759 (+ 1491us) client_negotiation.cc:240] Negotiation successful
0807 18:28:34.368178 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11631,"mutex_wait_us":624,"thread_start_us":4384,"threads_started":1}
I20260807 18:28:34.368798 1732 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:36823
I20260807 18:28:34.369757 3101 heartbeater.cc:347] Connected to a master server at 127.27.1.252:33443
I20260807 18:28:34.369899 3101 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:34.370575 3101 heartbeater.cc:511] Master 127.27.1.252:33443 requested a full tablet report, sending...
I20260807 18:28:34.372422 2586 ts_manager.cc:194] Registered new tserver with Master: 3eb0a9e28caf420487cc3e2db20f96c3 (127.27.1.195:34445)
I20260807 18:28:34.376184 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.322311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:37827 (local address 127.27.1.195:39519)
0807 18:28:34.333239 (+ 10928us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:34.333266 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:34.333521 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:34.336939 (+ 3418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:34.336947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:34.337883 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:34.338479 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.338497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.340586 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.340596 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.343377 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.343396 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.356989 (+ 13593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.357000 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.357262 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.359300 (+ 2038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:34.359327 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:34.365439 (+ 6112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:34.372770 (+ 7331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:34.372774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:34.372778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:34.373286 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:34.373861 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:34.373866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:34.373870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:34.374074 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:34.374900 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:34.374908 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:34.375113 (+ 205us) client_negotiation.cc:769] Sending connection context
0807 18:28:34.375305 (+ 192us) client_negotiation.cc:240] Negotiation successful
0807 18:28:34.375735 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10722,"mutex_wait_us":565,"spinlock_wait_cycles":94976,"thread_start_us":10846,"threads_started":1}
I20260807 18:28:34.377640 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.322631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:39519 (local address 127.27.1.253:37827)
0807 18:28:34.336178 (+ 13547us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.336189 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.336218 (+ 29us) server_negotiation.cc:407] Connection header received
0807 18:28:34.336296 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.336305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.336426 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.336577 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:34.338728 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.340357 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.343585 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.344386 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.355904 (+ 11518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.368088 (+ 12184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:34.368097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:34.368101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:34.368147 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:34.372602 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.373452 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.373458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.373461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.373664 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.374212 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.374217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.374220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.374646 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:34.374814 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.376759 (+ 1945us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.377081 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13346,"thread_start_us":14094,"threads_started":1}
I20260807 18:28:34.379082 3103 heartbeater.cc:347] Connected to a master server at 127.27.1.253:37827
I20260807 18:28:34.379256 3103 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:34.379746 3103 heartbeater.cc:511] Master 127.27.1.253:37827 requested a full tablet report, sending...
I20260807 18:28:34.381230 2383 ts_manager.cc:194] Registered new tserver with Master: 3eb0a9e28caf420487cc3e2db20f96c3 (127.27.1.195:34445)
I20260807 18:28:34.392477 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.344946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.27.1.253:37827)
0807 18:28:34.371709 (+ 26763us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.371715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.371734 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:28:34.371786 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.371795 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.371880 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.372003 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:34.375282 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.377007 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.379689 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.380631 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.380780 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.385513 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:34.385520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:34.385524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:34.385566 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:34.389170 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.390033 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.390039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.390043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.390211 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.390664 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.390670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.390673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.391028 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:34.391305 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.391779 (+ 474us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.392027 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26600,"thread_start_us":30104,"threads_started":1}
I20260807 18:28:34.400784 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.344998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.27.1.252:33443)
0807 18:28:34.363631 (+ 18633us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.363637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.384377 (+ 20740us) server_negotiation.cc:407] Connection header received
0807 18:28:34.384922 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.384928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.385027 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.385124 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:34.387607 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.388570 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.390414 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.391085 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.391204 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.394418 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:34.394423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:34.394426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:34.394457 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:34.397631 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.398385 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.398389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.398391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.398518 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.398960 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.398968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.398972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.399370 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:34.399625 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.400144 (+ 519us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.400403 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18501,"thread_start_us":18650,"threads_started":1}
I20260807 18:28:34.401268 3107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.345593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.27.1.254:38033)
0807 18:28:34.358144 (+ 12551us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.358153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.383428 (+ 25275us) server_negotiation.cc:407] Connection header received
0807 18:28:34.383548 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.383556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.383668 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.383804 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:34.386545 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.387659 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.389827 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.390479 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.390622 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.394686 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:34.394699 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:34.394703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:34.394737 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:34.398355 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.399222 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.399227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.399230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.399378 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.399721 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.399724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.399726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.400030 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:34.400249 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.400627 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.400872 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12412,"thread_start_us":37114,"threads_started":1}
I20260807 18:28:34.405124 1732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:28:34.412372 1732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:28:34.467769 3136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.431971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:39689 (local address 127.0.0.1:45882)
0807 18:28:34.456961 (+ 24990us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:34.456987 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:34.457164 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:34.458435 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:34.458446 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:34.458474 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:34.458917 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.458934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.461794 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.461803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.464571 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.464584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.466834 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.466841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.467032 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.467068 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 18:28:34.467189 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 18:28:34.467345 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24776,"thread_start_us":24576,"threads_started":1}
I20260807 18:28:34.467911 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.432678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.27.1.194:39689)
0807 18:28:34.456207 (+ 23529us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.456220 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.457343 (+ 1123us) server_negotiation.cc:407] Connection header received
0807 18:28:34.457522 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.457536 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.457923 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.458168 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:34.459200 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.461594 (+ 2394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.464726 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.466636 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.466807 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.467123 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.467264 (+ 141us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.467490 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23286,"thread_start_us":23132,"threads_started":1}
I20260807 18:28:34.471568 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.439659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35689 (local address 127.0.0.1:54154)
0807 18:28:34.462445 (+ 22786us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:34.462465 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:34.462587 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:34.463941 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:34.463948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:34.463982 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:34.464491 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.464505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.466592 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.466598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.468870 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.468884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.470629 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.470637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.470848 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.470890 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 18:28:34.471006 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 18:28:34.471174 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22628,"thread_start_us":14968,"threads_started":1}
I20260807 18:28:34.471604 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.440424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.27.1.193:35689)
0807 18:28:34.455738 (+ 15314us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.455752 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.463228 (+ 7476us) server_negotiation.cc:407] Connection header received
0807 18:28:34.463348 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.463357 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.463613 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.464723 (+ 1110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:34.465039 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.466442 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.469033 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.470409 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.470569 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.470791 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.471057 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.471262 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15052,"thread_start_us":14904,"threads_started":1}
I20260807 18:28:34.472121 2870 tablet_service.cc:1467] Processing CreateTablet for tablet b7a0c41fca5c48c7bc2cd555e3ef3fed (DEFAULT_TABLE table=test-table [id=fcbff6f02c4d4f6cbe075a33b9e5ddf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:34.472712 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.431573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:34445 (local address 127.0.0.1:42956)
0807 18:28:34.455044 (+ 23471us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:34.455073 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:34.456211 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:34.463753 (+ 7542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:34.463757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:34.463800 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:34.464346 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.464359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.467244 (+ 2885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.467266 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.469780 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.469798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.472072 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.472077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.472210 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.472233 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 18:28:34.472314 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 18:28:34.472413 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23258,"mutex_wait_us":1,"thread_start_us":30517,"threads_started":1}
I20260807 18:28:34.472890 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.431861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.27.1.195:34445)
0807 18:28:34.463231 (+ 31370us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.463238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.463268 (+ 30us) server_negotiation.cc:407] Connection header received
0807 18:28:34.463346 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.463353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.463552 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.463767 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:34.464583 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.466477 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.469968 (+ 3491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.471407 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.471561 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.471742 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.472394 (+ 652us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.472560 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31145,"thread_start_us":30929,"threads_started":1}
I20260807 18:28:34.476159 3029 tablet_service.cc:1467] Processing CreateTablet for tablet b7a0c41fca5c48c7bc2cd555e3ef3fed (DEFAULT_TABLE table=test-table [id=fcbff6f02c4d4f6cbe075a33b9e5ddf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:34.476241 2712 tablet_service.cc:1467] Processing CreateTablet for tablet b7a0c41fca5c48c7bc2cd555e3ef3fed (DEFAULT_TABLE table=test-table [id=fcbff6f02c4d4f6cbe075a33b9e5ddf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:34.477376 3029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a0c41fca5c48c7bc2cd555e3ef3fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:34.477511 2712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a0c41fca5c48c7bc2cd555e3ef3fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:34.480417 2870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a0c41fca5c48c7bc2cd555e3ef3fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:34.490664 3141 tablet_bootstrap.cc:492] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: Bootstrap starting.
I20260807 18:28:34.494045 3141 tablet_bootstrap.cc:654] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:34.495014 3142 tablet_bootstrap.cc:492] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: Bootstrap starting.
I20260807 18:28:34.495627 3141 log.cc:824] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:34.497013 3143 tablet_bootstrap.cc:492] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: Bootstrap starting.
I20260807 18:28:34.499883 3142 tablet_bootstrap.cc:654] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:34.500730 3141 tablet_bootstrap.cc:492] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: No bootstrap required, opened a new log
I20260807 18:28:34.500737 3143 tablet_bootstrap.cc:654] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:34.501055 3141 ts_tablet_manager.cc:1397] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260807 18:28:34.501264 3142 log.cc:824] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:34.501922 3143 log.cc:824] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:34.504738 3142 tablet_bootstrap.cc:492] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: No bootstrap required, opened a new log
I20260807 18:28:34.505016 3142 ts_tablet_manager.cc:1397] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 18:28:34.505213 3143 tablet_bootstrap.cc:492] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: No bootstrap required, opened a new log
I20260807 18:28:34.505460 3143 ts_tablet_manager.cc:1397] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260807 18:28:34.512693 3143 raft_consensus.cc:348] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } }
I20260807 18:28:34.513073 3143 raft_consensus.cc:374] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:34.513183 3143 raft_consensus.cc:729] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698923a8cfda4e21a16dbf1b1239f5bf, State: Initialized, Role: FOLLOWER
I20260807 18:28:34.513015 3141 raft_consensus.cc:348] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } }
I20260807 18:28:34.513440 3141 raft_consensus.cc:374] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:34.513548 3141 raft_consensus.cc:729] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf04659d5b944b2ab47b4a7d0f8e918, State: Initialized, Role: FOLLOWER
I20260807 18:28:34.513902 3143 consensus_queue.cc:260] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } }
I20260807 18:28:34.514940 3141 consensus_queue.cc:260] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } }
I20260807 18:28:34.516409 3143 ts_tablet_manager.cc:1428] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260807 18:28:34.516745 2940 heartbeater.cc:503] Master 127.27.1.254:38033 was elected leader, sending a full tablet report...
I20260807 18:28:34.517041 3142 raft_consensus.cc:348] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } }
I20260807 18:28:34.517509 3142 raft_consensus.cc:374] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:34.517619 3142 raft_consensus.cc:729] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb0a9e28caf420487cc3e2db20f96c3, State: Initialized, Role: FOLLOWER
I20260807 18:28:34.518460 3142 consensus_queue.cc:260] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } }
I20260807 18:28:34.519337 3141 ts_tablet_manager.cc:1428] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: Time spent starting tablet: real 0.018s user 0.018s sys 0.001s
I20260807 18:28:34.527649 3102 heartbeater.cc:503] Master 127.27.1.254:38033 was elected leader, sending a full tablet report...
I20260807 18:28:34.528460 3142 ts_tablet_manager.cc:1428] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: Time spent starting tablet: real 0.023s user 0.015s sys 0.009s
W20260807 18:28:34.546375 2785 tablet.cc:2369] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:28:34.554198 3149 raft_consensus.cc:482] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:34.554453 3149 raft_consensus.cc:504] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } }
I20260807 18:28:34.556218 3149 leader_election.cc:288] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 698923a8cfda4e21a16dbf1b1239f5bf (127.27.1.194:39689), ebf04659d5b944b2ab47b4a7d0f8e918 (127.27.1.193:35689)
I20260807 18:28:34.562318 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.556499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:39689 (local address 127.27.1.195:54045)
0807 18:28:34.556822 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:34.556841 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:34.556969 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:34.557395 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:34.557400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:34.557431 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:34.557642 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.557653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.558948 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.558956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.560369 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.560381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.561548 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.561554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.561765 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.561789 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 18:28:34.561899 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 18:28:34.562033 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260807 18:28:34.562318 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.556665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:54045 (local address 127.27.1.194:39689)
0807 18:28:34.556862 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.556869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.556909 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:28:34.557145 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.557151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.557236 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.557378 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:34.557890 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.558811 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.560556 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.561363 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.561534 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.561608 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.561941 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.562063 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 18:28:34.563310 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.556684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35689 (local address 127.27.1.195:50347)
0807 18:28:34.556935 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:34.556949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:34.557055 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:34.557594 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:34.557599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:34.557628 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:34.557859 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.557873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.559170 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.559176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.560852 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:34.560865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.562394 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.562404 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:34.562687 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.562735 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 18:28:34.562836 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 18:28:34.562999 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260807 18:28:34.563427 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a0c41fca5c48c7bc2cd555e3ef3fed" candidate_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" is_pre_election: true
I20260807 18:28:34.563776 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.556665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:50347 (local address 127.27.1.193:35689)
0807 18:28:34.556862 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.556881 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.557033 (+ 152us) server_negotiation.cc:407] Connection header received
0807 18:28:34.557235 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.557241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.557314 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.557406 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:34.558024 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.558977 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.561024 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.561998 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.562180 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.562278 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.562953 (+ 675us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.563091 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 18:28:34.564041 2891 raft_consensus.cc:2454] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb0a9e28caf420487cc3e2db20f96c3 in term 0.
I20260807 18:28:34.564646 2733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a0c41fca5c48c7bc2cd555e3ef3fed" candidate_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" is_pre_election: true
I20260807 18:28:34.564772 2977 leader_election.cc:302] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb0a9e28caf420487cc3e2db20f96c3, 698923a8cfda4e21a16dbf1b1239f5bf; no voters:
I20260807 18:28:34.565104 2733 raft_consensus.cc:2454] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb0a9e28caf420487cc3e2db20f96c3 in term 0.
I20260807 18:28:34.565210 3149 raft_consensus.cc:2781] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:28:34.565411 3149 raft_consensus.cc:482] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:28:34.565548 3149 raft_consensus.cc:3037] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:34.568181 3149 raft_consensus.cc:504] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } }
I20260807 18:28:34.568967 3149 leader_election.cc:288] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [CANDIDATE]: Term 1 election: Requested vote from peers 698923a8cfda4e21a16dbf1b1239f5bf (127.27.1.194:39689), ebf04659d5b944b2ab47b4a7d0f8e918 (127.27.1.193:35689)
I20260807 18:28:34.569298 2733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a0c41fca5c48c7bc2cd555e3ef3fed" candidate_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918"
I20260807 18:28:34.569501 2733 raft_consensus.cc:3037] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:34.571887 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a0c41fca5c48c7bc2cd555e3ef3fed" candidate_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698923a8cfda4e21a16dbf1b1239f5bf"
I20260807 18:28:34.571970 2733 raft_consensus.cc:2454] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb0a9e28caf420487cc3e2db20f96c3 in term 1.
I20260807 18:28:34.572085 2891 raft_consensus.cc:3037] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:34.572453 2976 leader_election.cc:302] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb0a9e28caf420487cc3e2db20f96c3, ebf04659d5b944b2ab47b4a7d0f8e918; no voters:
I20260807 18:28:34.572779 3149 raft_consensus.cc:2781] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:34.573272 3149 raft_consensus.cc:686] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 1 LEADER]: Becoming Leader. State: Replica: 3eb0a9e28caf420487cc3e2db20f96c3, State: Running, Role: LEADER
I20260807 18:28:34.573843 3149 consensus_queue.cc:237] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } }
I20260807 18:28:34.575091 2891 raft_consensus.cc:2454] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb0a9e28caf420487cc3e2db20f96c3 in term 1.
W20260807 18:28:34.576606 3105 tablet.cc:2369] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:28:34.579519 1731 catalog_manager.cc:5495] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb0a9e28caf420487cc3e2db20f96c3 (127.27.1.195). New cstate: current_term: 1 leader_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } health_report { overall_health: UNKNOWN } } }
I20260807 18:28:34.601826 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.596498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.27.1.195:34445)
0807 18:28:34.596699 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.596705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.596724 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:28:34.596895 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.596899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.596978 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.597131 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:34.597508 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.598327 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.599740 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.600135 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.600278 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.600763 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:34.601249 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:34.601405 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.601491 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.601570 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 18:28:34.610107 2891 raft_consensus.cc:1261] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf [term 1 FOLLOWER]: Refusing update from remote peer 3eb0a9e28caf420487cc3e2db20f96c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:34.610276 2733 raft_consensus.cc:1261] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918 [term 1 FOLLOWER]: Refusing update from remote peer 3eb0a9e28caf420487cc3e2db20f96c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:34.611253 3149 consensus_queue.cc:1035] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:34.611591 3150 consensus_queue.cc:1035] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:34.631841 3157 mvcc.cc:205] Tried to move back new op lower bound from 7315977480633188352 to 7315977480498864128. Current Snapshot: MvccSnapshot[applied={T|T < 7315977480633188352}]
I20260807 18:28:34.649909 1731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:34.650178 1731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:34.656888 1731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c98333801763413d99a0d26660b81d1a: Sending DeleteTablet for 3 replicas of tablet b7a0c41fca5c48c7bc2cd555e3ef3fed
I20260807 18:28:34.658697 3029 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a0c41fca5c48c7bc2cd555e3ef3fed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:34 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:42956
I20260807 18:28:34.658984 2870 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a0c41fca5c48c7bc2cd555e3ef3fed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:34 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:45882
I20260807 18:28:34.659587 2712 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a0c41fca5c48c7bc2cd555e3ef3fed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:34 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:54154
I20260807 18:28:34.660597 3167 tablet_replica.cc:330] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: stopping tablet replica
I20260807 18:28:34.661134 3167 raft_consensus.cc:2229] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:34.661739 3167 raft_consensus.cc:2258] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:34.672128 3167 ts_tablet_manager.cc:1905] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:34.674183 3166 tablet_replica.cc:330] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: stopping tablet replica
I20260807 18:28:34.674626 3166 raft_consensus.cc:2229] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:28:34.675498 3166 raft_consensus.cc:2258] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:34.677933 3166 ts_tablet_manager.cc:1905] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:34.680822 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 1697
I20260807 18:28:34.681921 1767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:34.687559 3167 ts_tablet_manager.cc:1918] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:34.687734 3167 log.cc:1196] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/b7a0c41fca5c48c7bc2cd555e3ef3fed
I20260807 18:28:34.688308 3167 ts_tablet_manager.cc:1939] T b7a0c41fca5c48c7bc2cd555e3ef3fed P ebf04659d5b944b2ab47b4a7d0f8e918: Deleting consensus metadata
I20260807 18:28:34.690366 3168 tablet_replica.cc:330] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: stopping tablet replica
I20260807 18:28:34.690923 3168 raft_consensus.cc:2229] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:34.691674 3168 raft_consensus.cc:2258] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:34.694244 3168 ts_tablet_manager.cc:1905] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:34.698163 3166 ts_tablet_manager.cc:1918] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:34.698359 3166 log.cc:1196] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/b7a0c41fca5c48c7bc2cd555e3ef3fed
I20260807 18:28:34.699020 3166 ts_tablet_manager.cc:1939] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 3eb0a9e28caf420487cc3e2db20f96c3: Deleting consensus metadata
I20260807 18:28:34.702144 3168 ts_tablet_manager.cc:1918] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:34.702303 3168 log.cc:1196] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/b7a0c41fca5c48c7bc2cd555e3ef3fed
I20260807 18:28:34.702932 3168 ts_tablet_manager.cc:1939] T b7a0c41fca5c48c7bc2cd555e3ef3fed P 698923a8cfda4e21a16dbf1b1239f5bf: Deleting consensus metadata
I20260807 18:28:34.712046 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.664596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.27.1.253:37827)
0807 18:28:34.664845 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.664856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.679487 (+ 14631us) server_negotiation.cc:407] Connection header received
0807 18:28:34.681987 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.681995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.682095 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.682188 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:34.689680 (+ 7492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.690913 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.693018 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.693748 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.693847 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.705073 (+ 11226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:34.705081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:34.705087 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:34.705139 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:34.709165 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.709926 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.709932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.709935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.710076 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.710441 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.710447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.710450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.710830 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:34.711088 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.711471 (+ 383us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.711708 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 18:28:34.725293 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.680276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.27.1.252:33443)
0807 18:28:34.680468 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.680474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.712054 (+ 31580us) server_negotiation.cc:407] Connection header received
0807 18:28:34.712300 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.712306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.712400 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.712520 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:34.714291 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.715018 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.716304 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.716733 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.716865 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.719577 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:34.719583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:34.719587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:34.719621 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:34.722341 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.722946 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.722951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.722959 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.723089 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.723705 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.723710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.723714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.724097 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:34.724363 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.724717 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.724945 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:28:34.876482 1717 catalog_manager.cc:4841] TS 698923a8cfda4e21a16dbf1b1239f5bf (127.27.1.194:39689): tablet b7a0c41fca5c48c7bc2cd555e3ef3fed (table test-table [id=fcbff6f02c4d4f6cbe075a33b9e5ddf1]) successfully deleted
I20260807 18:28:34.876737 1718 catalog_manager.cc:4841] TS 3eb0a9e28caf420487cc3e2db20f96c3 (127.27.1.195:34445): tablet b7a0c41fca5c48c7bc2cd555e3ef3fed (table test-table [id=fcbff6f02c4d4f6cbe075a33b9e5ddf1]) successfully deleted
W20260807 18:28:34.876959 1718 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.1.195:34445, user_credentials={real_user=slave}} blocked reactor thread for 173917us
I20260807 18:28:34.877574 1716 catalog_manager.cc:4841] TS ebf04659d5b944b2ab47b4a7d0f8e918 (127.27.1.193:35689): tablet b7a0c41fca5c48c7bc2cd555e3ef3fed (table test-table [id=fcbff6f02c4d4f6cbe075a33b9e5ddf1]) successfully deleted
W20260807 18:28:34.877743 1716 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.1.193:35689, user_credentials={real_user=slave}} blocked reactor thread for 187462us
I20260807 18:28:34.878364 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 1697
I20260807 18:28:34.936309 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.911439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.27.1.252:33443)
0807 18:28:34.911677 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.911683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.918904 (+ 7221us) server_negotiation.cc:407] Connection header received
0807 18:28:34.919470 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.919476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.919583 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.919693 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:34.921935 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.923042 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.924793 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.925301 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.925462 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.929118 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:34.929123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:34.929126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:34.929170 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:34.932991 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.933905 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.933912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.933916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.934087 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.934568 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.934575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.934578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.934908 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:34.935272 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.935605 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.935832 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260807 18:28:34.937651 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:34.911319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.27.1.253:37827)
0807 18:28:34.911517 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:34.911524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:34.919223 (+ 7699us) server_negotiation.cc:407] Connection header received
0807 18:28:34.919904 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:34.919910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:34.920000 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:34.920109 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:34.922327 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.923172 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.925761 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:34.926475 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:34.926656 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:34.930029 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:34.930037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:34.930042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:34.930085 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:34.933748 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.935226 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.935232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.935236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.935387 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:34.935847 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:34.935853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:34.935857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:34.936228 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:34.936475 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:34.936881 (+ 406us) server_negotiation.cc:299] Negotiation successful
0807 18:28:34.937115 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 18:28:35.628945 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:35.628045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.193:50601)
0807 18:28:35.628577 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:35.628690 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":288,"threads_started":1}
W20260807 18:28:35.629388 2783 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:38033 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
I20260807 18:28:35.639076 3179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:35.638067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.195:39501)
0807 18:28:35.638599 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:35.638766 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":312,"threads_started":1}
W20260807 18:28:35.639431 3102 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:38033 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
I20260807 18:28:35.641220 3180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:35.640395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.194:51161)
0807 18:28:35.640882 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:35.641031 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":251,"threads_started":1}
W20260807 18:28:35.641623 2940 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:38033 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
I20260807 18:28:36.157542 3182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c98333801763413d99a0d26660b81d1a)
I20260807 18:28:36.157796 3182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } }
I20260807 18:28:36.157696 3181 raft_consensus.cc:482] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c98333801763413d99a0d26660b81d1a)
I20260807 18:28:36.158265 3181 raft_consensus.cc:504] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } }
I20260807 18:28:36.159703 3182 leader_election.cc:288] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c98333801763413d99a0d26660b81d1a (127.27.1.254:38033), eec35020298c422d8dfbbea46d5dd5cc (127.27.1.253:37827)
I20260807 18:28:36.160272 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.158994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46748)
0807 18:28:36.159659 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.159988 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":310,"threads_started":1}
I20260807 18:28:36.168347 3181 leader_election.cc:288] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c98333801763413d99a0d26660b81d1a (127.27.1.254:38033), 9bb886b775c34a0d964e7dd52eb2518d (127.27.1.252:33443)
I20260807 18:28:36.168897 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.161296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46752)
0807 18:28:36.168133 (+ 6837us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.168503 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6684,"threads_started":1}
W20260807 18:28:36.177284 2364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
I20260807 18:28:36.179276 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.178538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46764)
0807 18:28:36.178795 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.178995 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 18:28:36.180087 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.179425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46774)
0807 18:28:36.179641 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.179789 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 18:28:36.179855 3185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.160399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.27.1.253:37827)
0807 18:28:36.161020 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:36.161028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:36.167203 (+ 6175us) server_negotiation.cc:407] Connection header received
0807 18:28:36.170385 (+ 3182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:36.170394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:36.170493 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:36.170590 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:36.171110 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:36.172790 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:36.176493 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:36.177993 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:36.178094 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:36.178198 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:36.179277 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0807 18:28:36.179432 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":495,"thread_start_us":286,"threads_started":1}
W20260807 18:28:36.180191 2569 leader_election.cc:334] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c98333801763413d99a0d26660b81d1a (127.27.1.254:38033): Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
I20260807 18:28:36.180614 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.167567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33443 (local address 127.0.0.1:51364)
0807 18:28:36.169115 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.169137 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:36.169292 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:36.171670 (+ 2378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:36.171678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:36.171715 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:36.173132 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:36.173144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:36.175043 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:36.175048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:36.177525 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:36.177543 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:36.179201 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:36.179210 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:36.179430 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:36.179462 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 18:28:36.179540 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 18:28:36.180286 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1434,"mutex_wait_us":1067,"thread_start_us":257,"threads_started":1}
W20260807 18:28:36.180680 2364 leader_election.cc:334] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c98333801763413d99a0d26660b81d1a (127.27.1.254:38033): Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
I20260807 18:28:36.180941 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.160103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:37827 (local address 127.0.0.1:39450)
0807 18:28:36.167076 (+ 6973us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.167099 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:36.170197 (+ 3098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:36.170607 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:36.170611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:36.170637 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:36.170920 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:36.170937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:36.172928 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:36.172933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:36.175376 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:36.175395 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:36.178779 (+ 3384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:36.178785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:36.179007 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:36.179084 (+ 77us) client_negotiation.cc:769] Sending connection context
0807 18:28:36.180337 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0807 18:28:36.180494 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6857,"mutex_wait_us":4,"thread_start_us":16381,"threads_started":1}
I20260807 18:28:36.181031 3189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.167799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.27.1.252:33443)
0807 18:28:36.170383 (+ 2584us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:36.170393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:36.170425 (+ 32us) server_negotiation.cc:407] Connection header received
0807 18:28:36.170515 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:36.170523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:36.170614 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:36.171924 (+ 1310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:36.173292 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:36.174904 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:36.177728 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:36.178940 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:36.179723 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:36.179838 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:36.179896 (+ 58us) server_negotiation.cc:299] Negotiation successful
0807 18:28:36.180037 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":667,"server-negotiator.queue_time_us":2378,"thread_start_us":2188,"threads_started":1}
I20260807 18:28:36.181537 2393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bb886b775c34a0d964e7dd52eb2518d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eec35020298c422d8dfbbea46d5dd5cc" is_pre_election: true
I20260807 18:28:36.181862 2393 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bb886b775c34a0d964e7dd52eb2518d in term 1.
I20260807 18:28:36.181820 2596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eec35020298c422d8dfbbea46d5dd5cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9bb886b775c34a0d964e7dd52eb2518d" is_pre_election: true
I20260807 18:28:36.182219 2596 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eec35020298c422d8dfbbea46d5dd5cc in term 1.
I20260807 18:28:36.182350 2571 leader_election.cc:302] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bb886b775c34a0d964e7dd52eb2518d, eec35020298c422d8dfbbea46d5dd5cc; no voters: c98333801763413d99a0d26660b81d1a
I20260807 18:28:36.182703 3182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 18:28:36.182715 2368 leader_election.cc:302] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bb886b775c34a0d964e7dd52eb2518d, eec35020298c422d8dfbbea46d5dd5cc; no voters: c98333801763413d99a0d26660b81d1a
I20260807 18:28:36.182821 3182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Starting leader election (detected failure of leader c98333801763413d99a0d26660b81d1a)
I20260807 18:28:36.182910 3182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 1 FOLLOWER]: Advancing to term 2
I20260807 18:28:36.183099 3181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 18:28:36.183254 3181 raft_consensus.cc:482] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader c98333801763413d99a0d26660b81d1a)
I20260807 18:28:36.183369 3181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 1 FOLLOWER]: Advancing to term 2
I20260807 18:28:36.185159 3182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } }
I20260807 18:28:36.185555 3181 raft_consensus.cc:504] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } }
I20260807 18:28:36.185787 3182 leader_election.cc:288] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 2 election: Requested vote from peers c98333801763413d99a0d26660b81d1a (127.27.1.254:38033), eec35020298c422d8dfbbea46d5dd5cc (127.27.1.253:37827)
I20260807 18:28:36.186269 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.185832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46780)
0807 18:28:36.186047 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.186128 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260807 18:28:36.186275 3181 leader_election.cc:288] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [CANDIDATE]: Term 2 election: Requested vote from peers c98333801763413d99a0d26660b81d1a (127.27.1.254:38033), 9bb886b775c34a0d964e7dd52eb2518d (127.27.1.252:33443)
I20260807 18:28:36.186326 2393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bb886b775c34a0d964e7dd52eb2518d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eec35020298c422d8dfbbea46d5dd5cc"
I20260807 18:28:36.186612 2393 raft_consensus.cc:2379] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bb886b775c34a0d964e7dd52eb2518d in current term 2: Already voted for candidate eec35020298c422d8dfbbea46d5dd5cc in this term.
I20260807 18:28:36.186738 2596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eec35020298c422d8dfbbea46d5dd5cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9bb886b775c34a0d964e7dd52eb2518d"
I20260807 18:28:36.187050 2596 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eec35020298c422d8dfbbea46d5dd5cc in current term 2: Already voted for candidate 9bb886b775c34a0d964e7dd52eb2518d in this term.
I20260807 18:28:36.187630 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.186783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46796)
0807 18:28:36.187430 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.187513 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528}
I20260807 18:28:36.187733 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.187293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46810)
0807 18:28:36.187466 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.187572 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 18:28:36.188038 2569 leader_election.cc:334] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c98333801763413d99a0d26660b81d1a (127.27.1.254:38033): Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
I20260807 18:28:36.188205 2569 leader_election.cc:302] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bb886b775c34a0d964e7dd52eb2518d; no voters: c98333801763413d99a0d26660b81d1a, eec35020298c422d8dfbbea46d5dd5cc
I20260807 18:28:36.188479 3182 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 18:28:36.188577 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.188193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:46816)
0807 18:28:36.188361 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.188454 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 18:28:36.188745 2364 leader_election.cc:334] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c98333801763413d99a0d26660b81d1a (127.27.1.254:38033): Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
I20260807 18:28:36.188844 2364 leader_election.cc:302] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eec35020298c422d8dfbbea46d5dd5cc; no voters: 9bb886b775c34a0d964e7dd52eb2518d, c98333801763413d99a0d26660b81d1a
I20260807 18:28:36.189071 3181 raft_consensus.cc:2726] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 18:28:36.630862 3192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.630051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.193:53483)
0807 18:28:36.630589 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.630675 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":302,"threads_started":1}
I20260807 18:28:36.640905 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.640063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.195:49239)
0807 18:28:36.640556 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.640694 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":310,"threads_started":1}
I20260807 18:28:36.643121 3194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:36.642276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.194:46707)
0807 18:28:36.642742 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:36.642873 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":255,"threads_started":1}
I20260807 18:28:37.632650 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:37.631766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.193:55229)
0807 18:28:37.632314 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:37.632449 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":277,"threads_started":1}
I20260807 18:28:37.642725 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:37.641802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.195:56141)
0807 18:28:37.642406 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:37.642516 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":378,"threads_started":1}
I20260807 18:28:37.644943 3197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:37.643987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.194:44933)
0807 18:28:37.644715 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:37.644791 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":523,"threads_started":1}
I20260807 18:28:38.090161 3198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:38.090354 3198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } }
I20260807 18:28:38.091046 3198 leader_election.cc:288] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c98333801763413d99a0d26660b81d1a (127.27.1.254:38033), eec35020298c422d8dfbbea46d5dd5cc (127.27.1.253:37827)
I20260807 18:28:38.091663 2393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bb886b775c34a0d964e7dd52eb2518d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eec35020298c422d8dfbbea46d5dd5cc" is_pre_election: true
I20260807 18:28:38.091961 2393 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bb886b775c34a0d964e7dd52eb2518d in term 2.
I20260807 18:28:38.092199 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.091288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:47086)
0807 18:28:38.091800 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.091954 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":261,"threads_started":1}
W20260807 18:28:38.092397 2569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260807 18:28:38.092499 2571 leader_election.cc:302] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bb886b775c34a0d964e7dd52eb2518d, eec35020298c422d8dfbbea46d5dd5cc; no voters:
I20260807 18:28:38.092818 3198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 18:28:38.092974 3198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:28:38.093113 3198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 2 FOLLOWER]: Advancing to term 3
I20260807 18:28:38.093878 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.093528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:47100)
0807 18:28:38.093678 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.093759 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 18:28:38.094050 2569 leader_election.cc:334] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c98333801763413d99a0d26660b81d1a (127.27.1.254:38033): Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
I20260807 18:28:38.095522 3198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } }
I20260807 18:28:38.096086 3198 leader_election.cc:288] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 3 election: Requested vote from peers c98333801763413d99a0d26660b81d1a (127.27.1.254:38033), eec35020298c422d8dfbbea46d5dd5cc (127.27.1.253:37827)
I20260807 18:28:38.096573 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.096192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:47110)
0807 18:28:38.096362 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.096451 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 18:28:38.096624 2393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bb886b775c34a0d964e7dd52eb2518d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eec35020298c422d8dfbbea46d5dd5cc"
I20260807 18:28:38.096854 2393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 2 FOLLOWER]: Advancing to term 3
I20260807 18:28:38.097559 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.097275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:47126)
0807 18:28:38.097411 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.097469 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 18:28:38.097761 2569 leader_election.cc:334] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c98333801763413d99a0d26660b81d1a (127.27.1.254:38033): Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
I20260807 18:28:38.099341 2393 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bb886b775c34a0d964e7dd52eb2518d in term 3.
I20260807 18:28:38.099730 2571 leader_election.cc:302] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bb886b775c34a0d964e7dd52eb2518d, eec35020298c422d8dfbbea46d5dd5cc; no voters: c98333801763413d99a0d26660b81d1a
I20260807 18:28:38.100028 3198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 18:28:38.100453 3198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [term 3 LEADER]: Becoming Leader. State: Replica: 9bb886b775c34a0d964e7dd52eb2518d, State: Running, Role: LEADER
I20260807 18:28:38.100828 3198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } }
I20260807 18:28:38.102576 3201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bb886b775c34a0d964e7dd52eb2518d. Latest consensus state: current_term: 3 leader_uuid: "9bb886b775c34a0d964e7dd52eb2518d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:38.102843 3201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:38.103621 3203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:38.107055 3203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:38.107929 3203 catalog_manager.cc:1259] Loaded cluster ID: 1bcd0fbcc44d441687caedeef987db55
I20260807 18:28:38.108036 3203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:38.109364 3203 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:38.110157 3203 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Loaded TSK: 0
I20260807 18:28:38.110967 3203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:28:38.121449 2586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:28:38.128451 2586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:28:38.132532 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.132082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:47132)
0807 18:28:38.132270 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.132389 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 18:28:38.132614 2393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [term 3 FOLLOWER]: Refusing update from remote peer 9bb886b775c34a0d964e7dd52eb2518d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260807 18:28:38.133450 3201 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260807 18:28:38.133694 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.133206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:47144)
0807 18:28:38.133445 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.133539 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260807 18:28:38.133867 2569 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d -> Peer c98333801763413d99a0d26660b81d1a (127.27.1.254:38033): Couldn't send request to peer c98333801763413d99a0d26660b81d1a. Status: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 18:28:38.136329 3205 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bb886b775c34a0d964e7dd52eb2518d. Latest consensus state: current_term: 3 leader_uuid: "9bb886b775c34a0d964e7dd52eb2518d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:38.136608 3205 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:28:38.137838 3198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9bb886b775c34a0d964e7dd52eb2518d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:38.138191 3198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:38.138345 3205 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9bb886b775c34a0d964e7dd52eb2518d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c98333801763413d99a0d26660b81d1a" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38033 } } peers { permanent_uuid: "eec35020298c422d8dfbbea46d5dd5cc" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 37827 } attrs { promote: false } } peers { permanent_uuid: "9bb886b775c34a0d964e7dd52eb2518d" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33443 } attrs { promote: false } } }
I20260807 18:28:38.138588 3205 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec35020298c422d8dfbbea46d5dd5cc [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:28:38.159427 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.146268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:39689 (local address 127.0.0.1:45896)
0807 18:28:38.147052 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.147070 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:38.147210 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:38.151976 (+ 4766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:38.151984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:38.152021 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:38.152518 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:38.152534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.154658 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.154667 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:38.157137 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:38.157149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.158477 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.158483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:38.158646 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.158701 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 18:28:38.158790 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 18:28:38.158959 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":2543,"threads_started":1}
I20260807 18:28:38.160338 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.147010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.27.1.194:39689)
0807 18:28:38.151533 (+ 4523us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:38.151543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:38.151569 (+ 26us) server_negotiation.cc:407] Connection header received
0807 18:28:38.151662 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:38.151671 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:38.151781 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:38.152744 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:38.152965 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.154452 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.157311 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.158287 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.158448 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.158545 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:38.159899 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0807 18:28:38.160067 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4365,"thread_start_us":4266,"threads_started":1}
I20260807 18:28:38.161021 2870 tablet_service.cc:1467] Processing CreateTablet for tablet e4878798a7784148805bc1e943834f27 (DEFAULT_TABLE table=test-table [id=e9ae8a3e4d9141879939e098b2d3da72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:38.161484 2870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4878798a7784148805bc1e943834f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:38.164296 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.145869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35689 (local address 127.0.0.1:54160)
0807 18:28:38.146029 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.146048 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:38.146177 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:38.157396 (+ 11219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:38.157403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:38.157429 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:38.157654 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:38.157663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.159801 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.159807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:38.161634 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:38.161649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.163548 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.163554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:38.163722 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.163753 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 18:28:38.163893 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 18:28:38.164026 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260807 18:28:38.164324 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.146591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.27.1.193:35689)
0807 18:28:38.157002 (+ 10411us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:38.157011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:38.157035 (+ 24us) server_negotiation.cc:407] Connection header received
0807 18:28:38.157120 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:38.157127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:38.157240 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:38.157362 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:38.157811 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.159676 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.161817 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.163371 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.163530 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.163631 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:38.163920 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 18:28:38.164055 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10214,"thread_start_us":10098,"threads_started":1}
I20260807 18:28:38.165210 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.146760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:34445 (local address 127.0.0.1:42980)
0807 18:28:38.156394 (+ 9634us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.156410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:38.156525 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:38.158723 (+ 2198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:38.158728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:38.158757 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:38.159179 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:38.159191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.160674 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.160680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:38.162676 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:38.162688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.164438 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.164444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:38.164610 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.164660 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 18:28:38.164754 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 18:28:38.164866 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9512,"thread_start_us":9322,"threads_started":1}
I20260807 18:28:38.165180 2712 tablet_service.cc:1467] Processing CreateTablet for tablet e4878798a7784148805bc1e943834f27 (DEFAULT_TABLE table=test-table [id=e9ae8a3e4d9141879939e098b2d3da72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:38.165601 2712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4878798a7784148805bc1e943834f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:38.165894 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.146882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.27.1.195:34445)
0807 18:28:38.158414 (+ 11532us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:38.158420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:38.158439 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:28:38.158515 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:38.158520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:38.158592 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:38.158700 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:38.159401 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.160486 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.162844 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.164081 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.164199 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.164298 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:38.165419 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0807 18:28:38.165571 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11407,"thread_start_us":11945,"threads_started":1}
I20260807 18:28:38.167922 3217 tablet_bootstrap.cc:492] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf: Bootstrap starting.
I20260807 18:28:38.169757 3217 tablet_bootstrap.cc:654] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:38.171922 3217 tablet_bootstrap.cc:492] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf: No bootstrap required, opened a new log
I20260807 18:28:38.172070 3217 ts_tablet_manager.cc:1397] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 18:28:38.172089 3029 tablet_service.cc:1467] Processing CreateTablet for tablet e4878798a7784148805bc1e943834f27 (DEFAULT_TABLE table=test-table [id=e9ae8a3e4d9141879939e098b2d3da72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:38.172538 3029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4878798a7784148805bc1e943834f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:38.172891 3217 raft_consensus.cc:348] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } }
I20260807 18:28:38.173166 3217 raft_consensus.cc:374] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:38.173272 3217 raft_consensus.cc:729] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698923a8cfda4e21a16dbf1b1239f5bf, State: Initialized, Role: FOLLOWER
I20260807 18:28:38.173544 3217 consensus_queue.cc:260] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } }
I20260807 18:28:38.180936 3217 ts_tablet_manager.cc:1428] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 18:28:38.181308 3224 tablet_bootstrap.cc:492] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3: Bootstrap starting.
I20260807 18:28:38.181322 3225 tablet_bootstrap.cc:492] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918: Bootstrap starting.
I20260807 18:28:38.181371 2939 heartbeater.cc:503] Master 127.27.1.252:33443 was elected leader, sending a full tablet report...
I20260807 18:28:38.182888 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.174637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.194:47595)
0807 18:28:38.182454 (+ 7817us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.182608 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7678,"threads_started":1}
I20260807 18:28:38.183279 3224 tablet_bootstrap.cc:654] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:38.183279 3225 tablet_bootstrap.cc:654] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:38.184980 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.184645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.194:36227)
0807 18:28:38.184798 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.184880 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 18:28:38.188467 3224 tablet_bootstrap.cc:492] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3: No bootstrap required, opened a new log
I20260807 18:28:38.188627 3224 ts_tablet_manager.cc:1397] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260807 18:28:38.188959 3225 tablet_bootstrap.cc:492] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918: No bootstrap required, opened a new log
I20260807 18:28:38.189127 3225 ts_tablet_manager.cc:1397] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260807 18:28:38.189515 3224 raft_consensus.cc:348] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } }
I20260807 18:28:38.189754 3224 raft_consensus.cc:374] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:38.189853 3224 raft_consensus.cc:729] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb0a9e28caf420487cc3e2db20f96c3, State: Initialized, Role: FOLLOWER
I20260807 18:28:38.189967 3225 raft_consensus.cc:348] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } }
I20260807 18:28:38.190223 3225 raft_consensus.cc:374] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:38.190124 3224 consensus_queue.cc:260] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } }
I20260807 18:28:38.190318 3225 raft_consensus.cc:729] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf04659d5b944b2ab47b4a7d0f8e918, State: Initialized, Role: FOLLOWER
I20260807 18:28:38.190604 3225 consensus_queue.cc:260] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } }
I20260807 18:28:38.191550 3225 ts_tablet_manager.cc:1428] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260807 18:28:38.191957 3224 ts_tablet_manager.cc:1428] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 18:28:38.193599 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.191571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.195:39177)
0807 18:28:38.193135 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.193293 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1440,"threads_started":1}
I20260807 18:28:38.195089 3101 heartbeater.cc:503] Master 127.27.1.252:33443 was elected leader, sending a full tablet report...
I20260807 18:28:38.195887 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.192372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.193:56711)
0807 18:28:38.195396 (+ 3024us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.195578 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2875,"thread_start_us":2743,"threads_started":1}
I20260807 18:28:38.197082 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.196719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.195:55043)
0807 18:28:38.196917 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.196982 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 18:28:38.296286 3219 raft_consensus.cc:482] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:38.296528 3219 raft_consensus.cc:504] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } }
I20260807 18:28:38.298149 3219 leader_election.cc:288] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebf04659d5b944b2ab47b4a7d0f8e918 (127.27.1.193:35689), 3eb0a9e28caf420487cc3e2db20f96c3 (127.27.1.195:34445)
I20260807 18:28:38.305514 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.298848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:52787 (local address 127.27.1.195:34445)
0807 18:28:38.299047 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:38.299052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:38.299068 (+ 16us) server_negotiation.cc:407] Connection header received
0807 18:28:38.299177 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:38.299184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:38.299253 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:38.299374 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:38.300070 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.301118 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.303560 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.304526 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.304675 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.304758 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:38.305109 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 18:28:38.305241 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 18:28:38.305509 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.298452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:35105 (local address 127.27.1.193:35689)
0807 18:28:38.298673 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:38.298678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:38.299666 (+ 988us) server_negotiation.cc:407] Connection header received
0807 18:28:38.299919 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:38.299925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:38.300018 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:38.300117 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:38.300536 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.301461 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.303805 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.304562 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.304751 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.304823 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:38.305153 (+ 330us) server_negotiation.cc:299] Negotiation successful
0807 18:28:38.305278 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 18:28:38.305533 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.298413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:34445 (local address 127.27.1.194:52787)
0807 18:28:38.298567 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.298589 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:38.298721 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:38.299512 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:38.299517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:38.299566 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:38.299921 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:38.299937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.301239 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.301243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:38.303413 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:38.303428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.304755 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.304762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:38.304952 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.304976 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 18:28:38.305049 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 18:28:38.305204 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":4}
I20260807 18:28:38.305678 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.298416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35689 (local address 127.27.1.194:35105)
0807 18:28:38.299558 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.299575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:38.299767 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:38.300139 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:38.300141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:38.300160 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:38.300387 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:38.300394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.301578 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.301583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:38.303649 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:38.303660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.304828 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.304834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:38.304974 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.304996 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 18:28:38.305111 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 18:28:38.305243 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":420,"thread_start_us":376,"threads_started":1}
I20260807 18:28:38.306304 2733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4878798a7784148805bc1e943834f27" candidate_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" is_pre_election: true
I20260807 18:28:38.306594 2733 raft_consensus.cc:2454] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698923a8cfda4e21a16dbf1b1239f5bf in term 0.
I20260807 18:28:38.306681 3049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4878798a7784148805bc1e943834f27" candidate_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" is_pre_election: true
I20260807 18:28:38.307178 2818 leader_election.cc:302] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698923a8cfda4e21a16dbf1b1239f5bf, ebf04659d5b944b2ab47b4a7d0f8e918; no voters:
I20260807 18:28:38.307178 3049 raft_consensus.cc:2454] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698923a8cfda4e21a16dbf1b1239f5bf in term 0.
I20260807 18:28:38.307614 3219 raft_consensus.cc:2781] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:28:38.307770 3219 raft_consensus.cc:482] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:28:38.307894 3219 raft_consensus.cc:3037] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:38.310237 3219 raft_consensus.cc:504] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } }
I20260807 18:28:38.310918 3219 leader_election.cc:288] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [CANDIDATE]: Term 1 election: Requested vote from peers ebf04659d5b944b2ab47b4a7d0f8e918 (127.27.1.193:35689), 3eb0a9e28caf420487cc3e2db20f96c3 (127.27.1.195:34445)
I20260807 18:28:38.311363 2733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4878798a7784148805bc1e943834f27" candidate_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918"
I20260807 18:28:38.311419 3049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4878798a7784148805bc1e943834f27" candidate_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb0a9e28caf420487cc3e2db20f96c3"
I20260807 18:28:38.311601 3049 raft_consensus.cc:3037] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:38.311594 2733 raft_consensus.cc:3037] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:38.314628 2733 raft_consensus.cc:2454] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698923a8cfda4e21a16dbf1b1239f5bf in term 1.
I20260807 18:28:38.314630 3049 raft_consensus.cc:2454] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698923a8cfda4e21a16dbf1b1239f5bf in term 1.
I20260807 18:28:38.315068 2820 leader_election.cc:302] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb0a9e28caf420487cc3e2db20f96c3, 698923a8cfda4e21a16dbf1b1239f5bf; no voters:
I20260807 18:28:38.315465 3219 raft_consensus.cc:2781] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:38.316089 3219 raft_consensus.cc:686] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 1 LEADER]: Becoming Leader. State: Replica: 698923a8cfda4e21a16dbf1b1239f5bf, State: Running, Role: LEADER
I20260807 18:28:38.317126 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.316638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.194:41481)
0807 18:28:38.316816 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.316930 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 18:28:38.317420 3219 consensus_queue.cc:237] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } } peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } }
I20260807 18:28:38.326478 2585 catalog_manager.cc:5495] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 698923a8cfda4e21a16dbf1b1239f5bf (127.27.1.194). New cstate: current_term: 1 leader_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "698923a8cfda4e21a16dbf1b1239f5bf" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 } health_report { overall_health: UNKNOWN } } }
I20260807 18:28:38.365746 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.360876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.27.1.194:39689)
0807 18:28:38.361070 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:38.361075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:38.361092 (+ 17us) server_negotiation.cc:407] Connection header received
0807 18:28:38.361237 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:38.361240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:38.361338 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:38.361480 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:38.361796 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.362505 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.363759 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.364185 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.364276 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.364641 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:38.365173 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:38.365285 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:38.365362 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 18:28:38.365440 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:28:38.370223 2733 raft_consensus.cc:1261] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918 [term 1 FOLLOWER]: Refusing update from remote peer 698923a8cfda4e21a16dbf1b1239f5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:38.370558 3049 raft_consensus.cc:1261] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3 [term 1 FOLLOWER]: Refusing update from remote peer 698923a8cfda4e21a16dbf1b1239f5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:38.370990 3233 consensus_queue.cc:1035] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf04659d5b944b2ab47b4a7d0f8e918" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:38.371475 3219 consensus_queue.cc:1035] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb0a9e28caf420487cc3e2db20f96c3" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 34445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:38.372895 2782 heartbeater.cc:503] Master 127.27.1.252:33443 was elected leader, sending a full tablet report...
I20260807 18:28:38.374240 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.373608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.193:39077)
0807 18:28:38.373899 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.374022 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260807 18:28:38.375001 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.374175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.195:58131)
0807 18:28:38.374632 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.374764 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285}
I20260807 18:28:38.380677 3236 mvcc.cc:205] Tried to move back new op lower bound from 7315977496036663296 to 7315977495832776704. Current Snapshot: MvccSnapshot[applied={T|T < 7315977496036663296}]
I20260807 18:28:38.381250 3238 mvcc.cc:205] Tried to move back new op lower bound from 7315977496036663296 to 7315977495832776704. Current Snapshot: MvccSnapshot[applied={T|T < 7315977496036663296}]
I20260807 18:28:38.383260 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.382718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.194:50539)
0807 18:28:38.382952 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.383070 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260807 18:28:38.382244 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.381653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.195:40977)
0807 18:28:38.381959 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.382077 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260807 18:28:38.391801 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.391051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.193:53089)
0807 18:28:38.391385 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.391545 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260807 18:28:38.393784 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.392915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.27.1.194:39491)
0807 18:28:38.393416 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.393526 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
I20260807 18:28:38.406299 2585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:38.406579 2585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:38.412671 2585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bb886b775c34a0d964e7dd52eb2518d: Sending DeleteTablet for 3 replicas of tablet e4878798a7784148805bc1e943834f27
I20260807 18:28:38.413609 2712 tablet_service.cc:1514] Processing DeleteTablet for tablet e4878798a7784148805bc1e943834f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:38 UTC) from {username='oryx', principal='oryx/127.27.1.252@KRBTEST.COM'} at 127.0.0.1:54160
I20260807 18:28:38.413862 3029 tablet_service.cc:1514] Processing DeleteTablet for tablet e4878798a7784148805bc1e943834f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:38 UTC) from {username='oryx', principal='oryx/127.27.1.252@KRBTEST.COM'} at 127.0.0.1:42980
I20260807 18:28:38.414721 2870 tablet_service.cc:1514] Processing DeleteTablet for tablet e4878798a7784148805bc1e943834f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:38 UTC) from {username='oryx', principal='oryx/127.27.1.252@KRBTEST.COM'} at 127.0.0.1:45896
I20260807 18:28:38.417368 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 2572
I20260807 18:28:38.420719 3249 tablet_replica.cc:330] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf: stopping tablet replica
I20260807 18:28:38.421066 3249 raft_consensus.cc:2229] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:28:38.421942 3249 raft_consensus.cc:2258] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:38.422540 3247 tablet_replica.cc:330] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918: stopping tablet replica
I20260807 18:28:38.422744 3247 raft_consensus.cc:2229] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:38.423024 3247 raft_consensus.cc:2258] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:38.424332 3249 ts_tablet_manager.cc:1905] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:38.424927 3247 ts_tablet_manager.cc:1905] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:38.425791 3248 tablet_replica.cc:330] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3: stopping tablet replica
I20260807 18:28:38.425992 3248 raft_consensus.cc:2229] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:38.426292 3248 raft_consensus.cc:2258] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:38.427573 3248 ts_tablet_manager.cc:1905] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:38.435182 3248 ts_tablet_manager.cc:1918] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:38.435400 3248 log.cc:1196] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/e4878798a7784148805bc1e943834f27
I20260807 18:28:38.436055 3248 ts_tablet_manager.cc:1939] T e4878798a7784148805bc1e943834f27 P 3eb0a9e28caf420487cc3e2db20f96c3: Deleting consensus metadata
I20260807 18:28:38.437377 2573 catalog_manager.cc:4841] TS 3eb0a9e28caf420487cc3e2db20f96c3 (127.27.1.195:34445): tablet e4878798a7784148805bc1e943834f27 (table test-table [id=e9ae8a3e4d9141879939e098b2d3da72]) successfully deleted
I20260807 18:28:38.438256 3249 ts_tablet_manager.cc:1918] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:38.438429 3249 log.cc:1196] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/e4878798a7784148805bc1e943834f27
I20260807 18:28:38.438880 3249 ts_tablet_manager.cc:1939] T e4878798a7784148805bc1e943834f27 P 698923a8cfda4e21a16dbf1b1239f5bf: Deleting consensus metadata
I20260807 18:28:38.439702 3247 ts_tablet_manager.cc:1918] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:38.439935 3247 log.cc:1196] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/e4878798a7784148805bc1e943834f27
I20260807 18:28:38.440398 3247 ts_tablet_manager.cc:1939] T e4878798a7784148805bc1e943834f27 P ebf04659d5b944b2ab47b4a7d0f8e918: Deleting consensus metadata
I20260807 18:28:38.440927 2571 catalog_manager.cc:4841] TS 698923a8cfda4e21a16dbf1b1239f5bf (127.27.1.194:39689): tablet e4878798a7784148805bc1e943834f27 (table test-table [id=e9ae8a3e4d9141879939e098b2d3da72]) successfully deleted
I20260807 18:28:38.441573 2570 catalog_manager.cc:4841] TS ebf04659d5b944b2ab47b4a7d0f8e918 (127.27.1.193:35689): tablet e4878798a7784148805bc1e943834f27 (table test-table [id=e9ae8a3e4d9141879939e098b2d3da72]) successfully deleted
I20260807 18:28:38.447613 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.418931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.27.1.193:35689)
0807 18:28:38.419138 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:38.419145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:38.429155 (+ 10010us) server_negotiation.cc:407] Connection header received
0807 18:28:38.429642 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:38.429650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:38.429748 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:38.429838 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:38.431880 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.433007 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.435017 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.435664 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.435766 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.439592 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:38.439615 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:38.439639 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:38.439709 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:38.444228 (+ 4519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:38.445016 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:38.445027 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:38.445030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:38.445208 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:38.445574 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:38.445579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:38.445583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:38.446096 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:38.446406 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:38.446785 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 18:28:38.447214 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:28:38.448318 2776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:38.544214 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 2572
I20260807 18:28:38.571893 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 2794
I20260807 18:28:38.588308 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.572724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.27.1.194:39689)
0807 18:28:38.572957 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:38.572963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:38.572987 (+ 24us) server_negotiation.cc:407] Connection header received
0807 18:28:38.573181 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:38.573186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:38.573290 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:38.573406 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:38.574747 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.575772 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.577149 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.577601 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.577736 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.580882 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:38.580901 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:38.580925 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:38.581001 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:38.585144 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:38.585986 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:38.585997 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:38.586001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:38.586197 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:38.586628 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:38.586633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:38.586637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:38.587087 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:38.587446 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:38.587707 (+ 261us) server_negotiation.cc:299] Negotiation successful
0807 18:28:38.587936 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:28:38.589227 2934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:38.591653 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.591220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:47156)
0807 18:28:38.591406 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.591504 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 18:28:38.592784 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.592342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38033 (local address 127.0.0.1:47168)
0807 18:28:38.592578 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:38.592661 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260807 18:28:38.724097 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 2794
I20260807 18:28:38.748653 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 2954
I20260807 18:28:38.761999 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:38.749506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42986 (local address 127.27.1.195:34445)
0807 18:28:38.749719 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:38.749724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:38.749741 (+ 17us) server_negotiation.cc:407] Connection header received
0807 18:28:38.749800 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:38.749805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:38.749898 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:38.749995 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:38.751441 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.752262 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.753476 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:38.753911 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:38.754018 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:38.756554 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:38.756572 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:38.756589 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:38.756637 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:38.759536 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:38.760173 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:38.760180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:38.760183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:38.760307 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:38.760641 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:38.760644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:38.760647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:38.761042 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:38.761348 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:38.761524 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 18:28:38.761695 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260807 18:28:38.762681 3096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:38.885497 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 2954
I20260807 18:28:38.912086 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 2232
I20260807 18:28:38.913358 2415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:39.034237 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 2232
I20260807 18:28:39.058780 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 2341
I20260807 18:28:39.059897 2618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:39.186177 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 2341
2026-08-07T18:28:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11756 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:28:39 dist-test-slave-1xrh krb5kdc[3259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:28:39 dist-test-slave-1xrh krb5kdc[3259](info): set up 2 sockets
Aug 07 18:28:39 dist-test-slave-1xrh krb5kdc[3259](info): commencing operation
krb5kdc: starting...
W20260807 18:28:40.347052 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:28:40 dist-test-slave-1xrh krb5kdc[3259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:28:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.254.keytab.
Entry for principal oryx/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.254.keytab.
I20260807 18:28:40.607015 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:43315
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.254.keytab
--principal=oryx/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:43315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:40.934762 3272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:40.935042 3272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:40.935261 3272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:40.942309 3272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:40.942406 3272 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 18:28:40.942498 3272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:40.942608 3272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:40.942688 3272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:40.942796 3272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:40.952683 3272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:43315
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.254.keytab
--principal=oryx/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:40.953982 3272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:40.955431 3272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:40.967367 3280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:40.967398 3278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:40.967597 3277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:40.968056 3272 server_base.cc:1034] running on GCE node
I20260807 18:28:40.968830 3272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:40.970150 3272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:40.971402 3272 hybrid_clock.cc:648] HybridClock initialized: now 1786127320971330 us; error 37 us; skew 500 ppm
I20260807 18:28:40.971728 3272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:40 dist-test-slave-1xrh krb5kdc[3259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127320, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:40.994004 3272 init.cc:377] Logged in from keytab as oryx/127.27.1.254@KRBTEST.COM (short username oryx)
I20260807 18:28:40.995828 3272 webserver.cc:460] Webserver started at http://127.27.1.254:41303/ using document root <none> and password file <none>
I20260807 18:28:40.996622 3272 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:40.996760 3272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:40.997117 3272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:41.000368 3272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "fccbd6b85f704cc8b7cbaf820fb4e820"
format_stamp: "Formatted at 2026-08-07 18:28:40 on dist-test-slave-1xrh"
server_key: "114e9f3dc4271b3ce8fabf85e705cd7a"
server_key_iv: "2a4eab376eae9a23988ae79b19c9069d"
server_key_version: "encryptionkey@0"
I20260807 18:28:41.001176 3272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "fccbd6b85f704cc8b7cbaf820fb4e820"
format_stamp: "Formatted at 2026-08-07 18:28:40 on dist-test-slave-1xrh"
server_key: "114e9f3dc4271b3ce8fabf85e705cd7a"
server_key_iv: "2a4eab376eae9a23988ae79b19c9069d"
server_key_version: "encryptionkey@0"
I20260807 18:28:41.006480 3272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:28:41.010201 3287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:41.011677 3272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 18:28:41.011863 3272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "fccbd6b85f704cc8b7cbaf820fb4e820"
format_stamp: "Formatted at 2026-08-07 18:28:40 on dist-test-slave-1xrh"
server_key: "114e9f3dc4271b3ce8fabf85e705cd7a"
server_key_iv: "2a4eab376eae9a23988ae79b19c9069d"
server_key_version: "encryptionkey@0"
I20260807 18:28:41.012084 3272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:41.048756 3272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:41.053205 3272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:41.053500 3272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:41.076576 3272 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:43315
I20260807 18:28:41.076686 3355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:43315 every 8 connection(s)
I20260807 18:28:41.078310 3272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:41.081314 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 3272
I20260807 18:28:41.081562 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:41.082558 27655 external_mini_cluster.cc:1428] Setting key 3b64b517ee0d3116c2d095afcd2fe750
I20260807 18:28:41.084091 3356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:41.101527 3356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820: Bootstrap starting.
Aug 07 18:28:41 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:41.105914 3356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:41.107296 3356 log.cc:824] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:41.110867 3356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820: No bootstrap required, opened a new log
I20260807 18:28:41.113240 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:41.084672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.27.1.254:43315)
0807 18:28:41.085556 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:41.085573 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:41.085627 (+ 54us) server_negotiation.cc:407] Connection header received
0807 18:28:41.088442 (+ 2815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:41.088490 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:41.089121 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:41.089671 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:41.092440 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:41.094855 (+ 2415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:41.096997 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:41.098111 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:41.098309 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:41.103399 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:41.103413 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:41.103439 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:41.103502 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:41.107689 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:41.108516 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:41.108522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:41.108531 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:41.108724 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:41.109236 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:41.109241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:41.109244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:41.109952 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:41.111222 (+ 1270us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:41.111518 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 18:28:41.111936 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":366,"threads_started":1}
I20260807 18:28:41.121657 3356 raft_consensus.cc:348] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccbd6b85f704cc8b7cbaf820fb4e820" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43315 } }
I20260807 18:28:41.122200 3356 raft_consensus.cc:374] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:41.122326 3356 raft_consensus.cc:729] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fccbd6b85f704cc8b7cbaf820fb4e820, State: Initialized, Role: FOLLOWER
I20260807 18:28:41.122946 3356 consensus_queue.cc:260] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccbd6b85f704cc8b7cbaf820fb4e820" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43315 } }
I20260807 18:28:41.123222 3356 raft_consensus.cc:388] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:41.123360 3356 raft_consensus.cc:482] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:41.123570 3356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:41.125813 3356 raft_consensus.cc:504] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccbd6b85f704cc8b7cbaf820fb4e820" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43315 } }
I20260807 18:28:41.126356 3356 leader_election.cc:302] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fccbd6b85f704cc8b7cbaf820fb4e820; no voters:
I20260807 18:28:41.127255 3356 leader_election.cc:288] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:41.127389 3361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:41.128181 3361 raft_consensus.cc:686] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [term 1 LEADER]: Becoming Leader. State: Replica: fccbd6b85f704cc8b7cbaf820fb4e820, State: Running, Role: LEADER
I20260807 18:28:41.128746 3361 consensus_queue.cc:237] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccbd6b85f704cc8b7cbaf820fb4e820" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43315 } }
I20260807 18:28:41.129519 3356 sys_catalog.cc:564] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:41.131807 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fccbd6b85f704cc8b7cbaf820fb4e820. Latest consensus state: current_term: 1 leader_uuid: "fccbd6b85f704cc8b7cbaf820fb4e820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccbd6b85f704cc8b7cbaf820fb4e820" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43315 } } }
I20260807 18:28:41.132300 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:41.132929 3362 sys_catalog.cc:455] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fccbd6b85f704cc8b7cbaf820fb4e820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccbd6b85f704cc8b7cbaf820fb4e820" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43315 } } }
I20260807 18:28:41.133205 3369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:41.133209 3362 sys_catalog.cc:458] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:41.144209 3369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:41.157004 3369 catalog_manager.cc:1347] Generated new cluster ID: 7df54fa19f854485a872d1cc291e5237
I20260807 18:28:41.157124 3369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:41.169938 3369 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:41.171780 3369 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:41.182762 3369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820: Generated new TSK 0
I20260807 18:28:41.184011 3369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab.
Entry for principal oryx/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab.
I20260807 18:28:41.250597 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:43315
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab
--principal=oryx/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:41.589960 3386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:41.590278 3386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:41.590454 3386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:41.597563 3386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:41.597720 3386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:41.597886 3386 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 18:28:41.598031 3386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:41.607684 3386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab
--principal=oryx/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.1.254:43315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:41.609193 3386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:41.610581 3386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:41.624084 3391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:41.624028 3394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:41.623997 3392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:41.625094 3386 server_base.cc:1034] running on GCE node
I20260807 18:28:41.625699 3386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:41.626698 3386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:41.627903 3386 hybrid_clock.cc:648] HybridClock initialized: now 1786127321627820 us; error 61 us; skew 500 ppm
I20260807 18:28:41.628202 3386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:41 dist-test-slave-1xrh krb5kdc[3259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127321, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:41.631508 3386 init.cc:377] Logged in from keytab as oryx/127.27.1.193@KRBTEST.COM (short username oryx)
I20260807 18:28:41.632908 3386 webserver.cc:460] Webserver started at http://127.27.1.193:35139/ using document root <none> and password file <none>
I20260807 18:28:41.633677 3386 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:41.633800 3386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:41.634156 3386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:41.637095 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "56db8696eb2248fb927c7c02e8c60424"
format_stamp: "Formatted at 2026-08-07 18:28:41 on dist-test-slave-1xrh"
server_key: "90eb6d504509dbef7218a826a970c187"
server_key_iv: "f9f53474dff3dada2b64b076f724c8cc"
server_key_version: "encryptionkey@0"
I20260807 18:28:41.637881 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "56db8696eb2248fb927c7c02e8c60424"
format_stamp: "Formatted at 2026-08-07 18:28:41 on dist-test-slave-1xrh"
server_key: "90eb6d504509dbef7218a826a970c187"
server_key_iv: "f9f53474dff3dada2b64b076f724c8cc"
server_key_version: "encryptionkey@0"
I20260807 18:28:41.642832 3386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 18:28:41.646246 3403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:41.647657 3386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:41.647819 3386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "56db8696eb2248fb927c7c02e8c60424"
format_stamp: "Formatted at 2026-08-07 18:28:41 on dist-test-slave-1xrh"
server_key: "90eb6d504509dbef7218a826a970c187"
server_key_iv: "f9f53474dff3dada2b64b076f724c8cc"
server_key_version: "encryptionkey@0"
I20260807 18:28:41.648039 3386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:41.679245 3386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:41.683657 3386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:41.683961 3386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:41.686775 3386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:41.686895 3386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:41.687072 3386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:41.687167 3386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 07 18:28:41 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127321, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:41.714363 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:41.687953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:43315 (local address 127.27.1.193:58689)
0807 18:28:41.688823 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:41.688862 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:41.690760 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:41.691695 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:41.691708 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:41.692681 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:41.693806 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:41.693838 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:41.695260 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:41.695269 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:41.698327 (+ 3058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:41.698364 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:41.699375 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:41.699390 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:41.699758 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:41.700693 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:41.700733 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:41.704301 (+ 3568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:41.708940 (+ 4639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:41.708951 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:41.708970 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:41.709753 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:41.710290 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:41.710295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:41.710298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:41.710657 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:41.711448 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:41.711456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:41.711908 (+ 452us) client_negotiation.cc:769] Sending connection context
0807 18:28:41.712239 (+ 331us) client_negotiation.cc:240] Negotiation successful
0807 18:28:41.712866 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":412,"threads_started":1}
I20260807 18:28:41.715656 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:41.687246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:58689 (local address 127.27.1.254:43315)
0807 18:28:41.688146 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:41.688154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:41.688977 (+ 823us) server_negotiation.cc:407] Connection header received
0807 18:28:41.691143 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:41.691151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:41.691244 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:41.691382 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:41.694065 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:41.695053 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:41.698544 (+ 3491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:41.699051 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:41.699248 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:41.704581 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:41.704589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:41.704594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:41.704645 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:41.708712 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:41.709963 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:41.709970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:41.709973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:41.710133 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:41.710834 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:41.710837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:41.710839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:41.711278 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:41.711486 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:41.714923 (+ 3437us) server_negotiation.cc:299] Negotiation successful
0807 18:28:41.715240 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":668,"threads_started":1}
I20260807 18:28:41.731946 3386 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:34867
I20260807 18:28:41.731920 3532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:34867 every 8 connection(s)
I20260807 18:28:41.733445 3386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:41.735951 3533 heartbeater.cc:347] Connected to a master server at 127.27.1.254:43315
I20260807 18:28:41.736195 3533 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:41.736876 3533 heartbeater.cc:511] Master 127.27.1.254:43315 requested a full tablet report, sending...
I20260807 18:28:41.738085 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:41.722732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:41315 (local address 127.27.1.254:43315)
0807 18:28:41.722932 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:41.722939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:41.723872 (+ 933us) server_negotiation.cc:407] Connection header received
0807 18:28:41.724102 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:41.724108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:41.724204 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:41.724310 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:41.725858 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:41.726889 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:41.729170 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:41.729790 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:41.729974 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:41.732124 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:41.732129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:41.732132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:41.732187 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:41.735174 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:41.735952 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:41.735958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:41.735963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:41.736153 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:41.736554 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:41.736559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:41.736562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:41.736831 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:41.736994 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:41.737450 (+ 456us) server_negotiation.cc:299] Negotiation successful
0807 18:28:41.737670 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:28:41.738121 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:41.722768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:43315 (local address 127.27.1.193:41315)
0807 18:28:41.723741 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:41.723758 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:41.723956 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:41.724496 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:41.724500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:41.725174 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:41.725636 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:41.725658 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:41.727057 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:41.727063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:41.729003 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:41.729015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:41.729992 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:41.729999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:41.730173 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:41.730931 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:41.730963 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:41.731910 (+ 947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:41.735301 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:41.735306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:41.735310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:41.735821 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:41.736281 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:41.736287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:41.736290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:41.736432 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:41.736960 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:41.736965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:41.737190 (+ 225us) client_negotiation.cc:769] Sending connection context
0807 18:28:41.737332 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 18:28:41.737672 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":630,"threads_started":1}
I20260807 18:28:41.739037 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 3386
I20260807 18:28:41.739269 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:28:41.739320 3308 ts_manager.cc:194] Registered new tserver with Master: 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867)
I20260807 18:28:41.740083 27655 external_mini_cluster.cc:1428] Setting key bac1477a6f23f1c55832820c835aebad
I20260807 18:28:41.740958 3308 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:58689
WARNING: no policy specified for oryx/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab.
Entry for principal oryx/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab.
I20260807 18:28:41.798731 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:43315
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab
--principal=oryx/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:42.134167 3537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:42.134433 3537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:42.134603 3537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:42.141638 3537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:42.141733 3537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:42.141814 3537 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 18:28:42.141968 3537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:42.151724 3537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab
--principal=oryx/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.1.254:43315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:42.153002 3537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:42.154317 3537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:42.167806 3542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:42.167742 3543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:42.167778 3545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:42.169133 3537 server_base.cc:1034] running on GCE node
I20260807 18:28:42.169727 3537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:42.170642 3537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:42.171856 3537 hybrid_clock.cc:648] HybridClock initialized: now 1786127322171798 us; error 22 us; skew 500 ppm
I20260807 18:28:42.172155 3537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:42 dist-test-slave-1xrh krb5kdc[3259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127322, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:42.175343 3537 init.cc:377] Logged in from keytab as oryx/127.27.1.194@KRBTEST.COM (short username oryx)
I20260807 18:28:42.176842 3537 webserver.cc:460] Webserver started at http://127.27.1.194:41937/ using document root <none> and password file <none>
I20260807 18:28:42.177551 3537 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:42.177666 3537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:42.178021 3537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:42.180996 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "5b423204c1fc426e9470b84a0d35d458"
format_stamp: "Formatted at 2026-08-07 18:28:42 on dist-test-slave-1xrh"
server_key: "52c2cfb56553d7afce27e9ad5b51c87f"
server_key_iv: "6b8867b01e3fd55fdc69963bafd94ac5"
server_key_version: "encryptionkey@0"
I20260807 18:28:42.181793 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "5b423204c1fc426e9470b84a0d35d458"
format_stamp: "Formatted at 2026-08-07 18:28:42 on dist-test-slave-1xrh"
server_key: "52c2cfb56553d7afce27e9ad5b51c87f"
server_key_iv: "6b8867b01e3fd55fdc69963bafd94ac5"
server_key_version: "encryptionkey@0"
I20260807 18:28:42.186867 3537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 18:28:42.190380 3552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:42.191752 3537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:42.191927 3537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "5b423204c1fc426e9470b84a0d35d458"
format_stamp: "Formatted at 2026-08-07 18:28:42 on dist-test-slave-1xrh"
server_key: "52c2cfb56553d7afce27e9ad5b51c87f"
server_key_iv: "6b8867b01e3fd55fdc69963bafd94ac5"
server_key_version: "encryptionkey@0"
I20260807 18:28:42.192144 3537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:42.207304 3537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:42.211181 3537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:42.211455 3537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:42.214254 3537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:42.214354 3537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:42.214552 3537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:42.214619 3537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 18:28:42 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127322, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:42.242878 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.214720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:45741 (local address 127.27.1.254:43315)
0807 18:28:42.214900 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:42.214906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:42.216496 (+ 1590us) server_negotiation.cc:407] Connection header received
0807 18:28:42.219624 (+ 3128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:42.219629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:42.219747 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:42.219874 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:42.222680 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.223999 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.227448 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.228250 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.228422 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.234326 (+ 5904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:42.234336 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:42.234340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:42.234384 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:42.238610 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:42.239724 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:42.239732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:42.239738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:42.239879 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:42.240561 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:42.240566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:42.240569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:42.240890 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:42.241071 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:42.242235 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0807 18:28:42.242490 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 18:28:42.244359 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.215403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:43315 (local address 127.27.1.194:45741)
0807 18:28:42.216343 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:42.216386 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:42.219250 (+ 2864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:42.220168 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:42.220184 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:42.221327 (+ 1143us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:42.222371 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.222398 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.224211 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.224224 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.227252 (+ 3028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.227272 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.228568 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.228581 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.228930 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.229924 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:42.229964 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:42.234042 (+ 4078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:42.238781 (+ 4739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:42.238792 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:42.238816 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:42.239520 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:42.240025 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:42.240033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:42.240044 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:42.240392 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:42.241172 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:42.241182 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:42.241695 (+ 513us) client_negotiation.cc:769] Sending connection context
0807 18:28:42.242104 (+ 409us) client_negotiation.cc:240] Negotiation successful
0807 18:28:42.242742 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":427,"threads_started":1}
I20260807 18:28:42.255940 3537 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:33043
I20260807 18:28:42.255913 3679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:33043 every 8 connection(s)
I20260807 18:28:42.257301 3537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:42.259922 3680 heartbeater.cc:347] Connected to a master server at 127.27.1.254:43315
I20260807 18:28:42.260242 3680 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:42.260998 3680 heartbeater.cc:511] Master 127.27.1.254:43315 requested a full tablet report, sending...
I20260807 18:28:42.262428 3308 ts_manager.cc:194] Registered new tserver with Master: 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043)
I20260807 18:28:42.263346 3308 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:45741
I20260807 18:28:42.264326 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 3537
I20260807 18:28:42.264508 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:28:42.264972 27655 external_mini_cluster.cc:1428] Setting key 78e8e59f4f79fd85e40dc387717be255
I20260807 18:28:42.275189 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.261911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:46265 (local address 127.27.1.254:43315)
0807 18:28:42.262108 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:42.262113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:42.262749 (+ 636us) server_negotiation.cc:407] Connection header received
0807 18:28:42.262951 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:42.262956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:42.263034 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:42.263153 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:42.264208 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.265319 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.266891 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.267364 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.267462 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.269755 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:42.269760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:42.269763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:42.269797 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:42.272442 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:42.273253 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:42.273257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:42.273259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:42.273408 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:42.273816 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:42.273820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:42.273823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:42.274126 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:42.274240 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:42.274729 (+ 489us) server_negotiation.cc:299] Negotiation successful
0807 18:28:42.274914 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:28:42.275650 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.261983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:43315 (local address 127.27.1.194:46265)
0807 18:28:42.262620 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:42.262654 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:42.262805 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:42.263200 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:42.263207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:42.263687 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:42.264054 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.264075 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.265475 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.265483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.266750 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.266763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.267549 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.267556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.267780 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.268465 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:42.268493 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:42.269569 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:42.272576 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:42.272583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:42.272590 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:42.273141 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:42.273520 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:42.273528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:42.273534 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:42.273697 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:42.274250 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:42.274258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:42.274463 (+ 205us) client_negotiation.cc:769] Sending connection context
0807 18:28:42.274624 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 18:28:42.274964 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":348,"threads_started":1}
WARNING: no policy specified for oryx/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab.
Entry for principal oryx/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab.
I20260807 18:28:42.322201 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:43315
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab
--principal=oryx/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:42.653677 3687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:42.653954 3687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:42.654050 3687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:42.661415 3687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:42.661521 3687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:42.661602 3687 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 18:28:42.661684 3687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:42.671309 3687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab
--principal=oryx/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.1.254:43315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:42.672523 3687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:42.673811 3687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:42.687237 3693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:42.687333 3695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:42.687309 3692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:42.688082 3687 server_base.cc:1034] running on GCE node
I20260807 18:28:42.688645 3687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:42.689481 3687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:42.690696 3687 hybrid_clock.cc:648] HybridClock initialized: now 1786127322690621 us; error 49 us; skew 500 ppm
I20260807 18:28:42.691021 3687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:42 dist-test-slave-1xrh krb5kdc[3259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127322, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:42.694275 3687 init.cc:377] Logged in from keytab as oryx/127.27.1.195@KRBTEST.COM (short username oryx)
I20260807 18:28:42.695837 3687 webserver.cc:460] Webserver started at http://127.27.1.195:35323/ using document root <none> and password file <none>
I20260807 18:28:42.696622 3687 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:42.696754 3687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:42.697088 3687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:42.700042 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "8f90ac215f4742d7adc29c062fadaa20"
format_stamp: "Formatted at 2026-08-07 18:28:42 on dist-test-slave-1xrh"
server_key: "588a3f36be4827a94e991b0d3c2aa46f"
server_key_iv: "47eec58ff245c8d3ec565b77cc9bbe30"
server_key_version: "encryptionkey@0"
I20260807 18:28:42.700834 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "8f90ac215f4742d7adc29c062fadaa20"
format_stamp: "Formatted at 2026-08-07 18:28:42 on dist-test-slave-1xrh"
server_key: "588a3f36be4827a94e991b0d3c2aa46f"
server_key_iv: "47eec58ff245c8d3ec565b77cc9bbe30"
server_key_version: "encryptionkey@0"
I20260807 18:28:42.705782 3687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:28:42.709551 3703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:42.710808 3687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:28:42.710979 3687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "8f90ac215f4742d7adc29c062fadaa20"
format_stamp: "Formatted at 2026-08-07 18:28:42 on dist-test-slave-1xrh"
server_key: "588a3f36be4827a94e991b0d3c2aa46f"
server_key_iv: "47eec58ff245c8d3ec565b77cc9bbe30"
server_key_version: "encryptionkey@0"
I20260807 18:28:42.711248 3687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:42.725137 3687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:42.729027 3687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:42.729301 3687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:42.732280 3687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:42.732407 3687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:42.732555 3687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:42.732643 3687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:42.744089 3533 heartbeater.cc:503] Master 127.27.1.254:43315 was elected leader, sending a full tablet report...
Aug 07 18:28:42 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127322, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:42.758414 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.732981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:34095 (local address 127.27.1.254:43315)
0807 18:28:42.733173 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:42.733180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:42.734887 (+ 1707us) server_negotiation.cc:407] Connection header received
0807 18:28:42.737922 (+ 3035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:42.737930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:42.738038 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:42.738170 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:42.739917 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.741299 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.743818 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.744574 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.744720 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.749747 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:42.749752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:42.749756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:42.749790 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:42.754097 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:42.754862 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:42.754869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:42.754875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:42.755068 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:42.755670 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:42.755675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:42.755679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:42.756107 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:42.756258 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:42.757520 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0807 18:28:42.757822 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:28:42.759411 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.733754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:43315 (local address 127.27.1.195:34095)
0807 18:28:42.734704 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:42.734762 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:42.737608 (+ 2846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:42.738368 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:42.738375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:42.738906 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:42.739670 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.739702 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.741524 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.741529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.743635 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.743650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.744888 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.744899 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.745310 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.746275 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:42.746299 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:42.749423 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:42.754260 (+ 4837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:42.754267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:42.754281 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:42.754727 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:42.755215 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:42.755221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:42.755227 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:42.755532 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:42.756226 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:42.756234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:42.756876 (+ 642us) client_negotiation.cc:769] Sending connection context
0807 18:28:42.757308 (+ 432us) client_negotiation.cc:240] Negotiation successful
0807 18:28:42.758018 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":435,"threads_started":1}
I20260807 18:28:42.782173 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.766417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:43315 (local address 127.27.1.195:44411)
0807 18:28:42.767598 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:42.767626 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:42.767742 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:42.768205 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:42.768211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:42.768900 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:42.769347 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.769361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.770348 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.770355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.771465 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.771507 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.772714 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.772727 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.772869 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.773613 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:42.773631 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:42.774847 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:42.779563 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:42.779571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:42.779577 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:42.780019 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:42.780481 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:42.780485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:42.780488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:42.780606 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:42.781309 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:42.781313 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:42.781470 (+ 157us) client_negotiation.cc:769] Sending connection context
0807 18:28:42.781586 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 18:28:42.781845 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":865,"threads_started":1}
I20260807 18:28:42.782560 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.766420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:44411 (local address 127.27.1.254:43315)
0807 18:28:42.766662 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:42.766671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:42.767725 (+ 1054us) server_negotiation.cc:407] Connection header received
0807 18:28:42.767845 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:42.767854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:42.767949 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:42.768057 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:42.769507 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.770204 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.771675 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.772475 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.772626 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.775091 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:42.775096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:42.775099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:42.775199 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:42.779401 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:42.780140 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:42.780149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:42.780155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:42.780347 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:42.780710 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:42.780718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:42.780723 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:42.781186 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:42.781380 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:42.781758 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 18:28:42.782066 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 18:28:42.793726 3687 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:45635
I20260807 18:28:42.793759 3832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:45635 every 8 connection(s)
I20260807 18:28:42.794955 3687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:42.797189 3833 heartbeater.cc:347] Connected to a master server at 127.27.1.254:43315
I20260807 18:28:42.797435 3833 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:42.798085 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 3687
I20260807 18:28:42.798267 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:28:42.798311 3833 heartbeater.cc:511] Master 127.27.1.254:43315 requested a full tablet report, sending...
I20260807 18:28:42.798861 27655 external_mini_cluster.cc:1428] Setting key 72a0151c94620d8364b3312716008e45
I20260807 18:28:42.799885 3308 ts_manager.cc:194] Registered new tserver with Master: 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635)
I20260807 18:28:42.800933 3308 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:34095
I20260807 18:28:42.802119 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 18:28:42 dist-test-slave-1xrh krb5kdc[3259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127322, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 18:28:42 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:42 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
W20260807 18:28:42.861805 3412 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:42.850776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.27.1.254:43315)
0807 18:28:42.851034 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:42.851043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:42.851352 (+ 309us) server_negotiation.cc:407] Connection header received
0807 18:28:42.851607 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:42.851611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:42.851695 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:42.851801 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:42.853230 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.853954 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.855211 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.855610 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.855775 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.861471 (+ 5696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48068: BlockingRecv error: recv got EOF from 127.0.0.1:48068 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 07 18:28:42 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127322, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:42.878573 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.865244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.27.1.254:43315)
0807 18:28:42.865502 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:42.865510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:42.865771 (+ 261us) server_negotiation.cc:407] Connection header received
0807 18:28:42.865997 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:42.866003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:42.866080 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:42.866181 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:42.867332 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.868050 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.869285 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.869739 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.869875 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.873459 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:42.873464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:42.873468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:42.873502 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:42.876012 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:42.876651 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:42.876657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:42.876661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:42.876801 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:42.877179 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:42.877184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:42.877188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:42.877461 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:42.877784 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:42.878023 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 18:28:42.878259 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 18:28:42.882797 3307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:28:42.889986 3307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:28:42.924788 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.902446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:45635 (local address 127.0.0.1:60210)
0807 18:28:42.915705 (+ 13259us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:42.915722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:42.915852 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:42.916799 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:42.916806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:42.916835 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:42.917538 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.917553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.919520 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.919525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.921879 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.921892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.923619 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.923627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.923863 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.923925 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 18:28:42.924085 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 18:28:42.924297 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13146,"mutex_wait_us":100,"thread_start_us":13001,"threads_started":1}
I20260807 18:28:42.925838 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.902547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.27.1.194:33043)
0807 18:28:42.914169 (+ 11622us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:42.914182 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:42.915190 (+ 1008us) server_negotiation.cc:407] Connection header received
0807 18:28:42.915325 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:42.915337 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:42.915639 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:42.915882 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:42.917389 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.918924 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.922595 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.923974 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.924132 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.924317 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:42.925466 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0807 18:28:42.925590 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11407,"thread_start_us":11253,"threads_started":1}
I20260807 18:28:42.926249 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.902089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34867 (local address 127.0.0.1:49898)
0807 18:28:42.914668 (+ 12579us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:42.914735 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:42.915086 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:42.916127 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:42.916132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:42.916158 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:42.916736 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.916752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.918987 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.918997 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.922427 (+ 3430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.922436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.925407 (+ 2971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.925417 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.925631 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.925657 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 18:28:42.925744 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 18:28:42.925844 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12416,"mutex_wait_us":287,"spinlock_wait_cycles":2944,"thread_start_us":9981,"threads_started":1}
I20260807 18:28:42.926299 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.903205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.27.1.193:34867)
0807 18:28:42.913009 (+ 9804us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:42.913026 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:42.914867 (+ 1841us) server_negotiation.cc:407] Connection header received
0807 18:28:42.915417 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:42.915428 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:42.915706 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:42.915911 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:42.917137 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.918834 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.923267 (+ 4433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.924875 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.925026 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.925228 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:42.925819 (+ 591us) server_negotiation.cc:299] Negotiation successful
0807 18:28:42.926000 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9084,"thread_start_us":8574,"threads_started":1}
I20260807 18:28:42.926333 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.902846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.27.1.195:45635)
0807 18:28:42.915140 (+ 12294us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:42.915150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:42.915841 (+ 691us) server_negotiation.cc:407] Connection header received
0807 18:28:42.916039 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:42.916046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:42.916268 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:42.916494 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:42.917778 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.919374 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.922061 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.923415 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.923553 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.923811 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:42.924138 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 18:28:42.924305 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12079,"thread_start_us":11885,"threads_started":1}
I20260807 18:28:42.927523 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:42.901939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:33043 (local address 127.0.0.1:38668)
0807 18:28:42.912513 (+ 10574us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:42.914876 (+ 2363us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:42.915073 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:42.915959 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:42.915967 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:42.915997 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:42.916623 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.916635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.919050 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.919061 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.922418 (+ 3357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:42.922435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:42.924664 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:42.924671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:42.924856 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:42.924888 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 18:28:42.924977 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 18:28:42.925111 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10253,"mutex_wait_us":427,"spinlock_wait_cycles":8064,"thread_start_us":12378,"threads_started":1}
I20260807 18:28:42.930472 3608 tablet_service.cc:1467] Processing CreateTablet for tablet 13c4ff71ce974982a34af2f2ef7dcaa5 (DEFAULT_TABLE table=test-table [id=0d717db9e5fe48a1b7d3173d5bfc7f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:42.930461 3458 tablet_service.cc:1467] Processing CreateTablet for tablet 13c4ff71ce974982a34af2f2ef7dcaa5 (DEFAULT_TABLE table=test-table [id=0d717db9e5fe48a1b7d3173d5bfc7f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:42.930461 3756 tablet_service.cc:1467] Processing CreateTablet for tablet 13c4ff71ce974982a34af2f2ef7dcaa5 (DEFAULT_TABLE table=test-table [id=0d717db9e5fe48a1b7d3173d5bfc7f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:42.931762 3608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c4ff71ce974982a34af2f2ef7dcaa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:42.931993 3458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c4ff71ce974982a34af2f2ef7dcaa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:42.932006 3756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c4ff71ce974982a34af2f2ef7dcaa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:42.944489 3859 tablet_bootstrap.cc:492] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: Bootstrap starting.
I20260807 18:28:42.948172 3860 tablet_bootstrap.cc:492] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: Bootstrap starting.
I20260807 18:28:42.948328 3859 tablet_bootstrap.cc:654] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:42.949652 3859 log.cc:824] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:42.951115 3861 tablet_bootstrap.cc:492] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: Bootstrap starting.
I20260807 18:28:42.952124 3860 tablet_bootstrap.cc:654] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:42.953373 3859 tablet_bootstrap.cc:492] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: No bootstrap required, opened a new log
I20260807 18:28:42.953517 3860 log.cc:824] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:42.953716 3859 ts_tablet_manager.cc:1397] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 18:28:42.956339 3861 tablet_bootstrap.cc:654] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:42.957885 3861 log.cc:824] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:42.960973 3861 tablet_bootstrap.cc:492] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: No bootstrap required, opened a new log
I20260807 18:28:42.961284 3861 ts_tablet_manager.cc:1397] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260807 18:28:42.961519 3860 tablet_bootstrap.cc:492] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: No bootstrap required, opened a new log
I20260807 18:28:42.961864 3860 ts_tablet_manager.cc:1397] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260807 18:28:42.964536 3859 raft_consensus.cc:348] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:42.965106 3859 raft_consensus.cc:374] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:42.965247 3859 raft_consensus.cc:729] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b423204c1fc426e9470b84a0d35d458, State: Initialized, Role: FOLLOWER
I20260807 18:28:42.965950 3859 consensus_queue.cc:260] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:42.967233 3680 heartbeater.cc:503] Master 127.27.1.254:43315 was elected leader, sending a full tablet report...
I20260807 18:28:42.968025 3859 ts_tablet_manager.cc:1428] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260807 18:28:42.970799 3861 raft_consensus.cc:348] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:42.971177 3861 raft_consensus.cc:374] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:42.971287 3861 raft_consensus.cc:729] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90ac215f4742d7adc29c062fadaa20, State: Initialized, Role: FOLLOWER
I20260807 18:28:42.971942 3861 consensus_queue.cc:260] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:42.973201 3833 heartbeater.cc:503] Master 127.27.1.254:43315 was elected leader, sending a full tablet report...
I20260807 18:28:42.973384 3860 raft_consensus.cc:348] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:42.973728 3861 ts_tablet_manager.cc:1428] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260807 18:28:42.973866 3860 raft_consensus.cc:374] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:42.973973 3860 raft_consensus.cc:729] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56db8696eb2248fb927c7c02e8c60424, State: Initialized, Role: FOLLOWER
I20260807 18:28:42.974597 3860 consensus_queue.cc:260] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:42.976574 3860 ts_tablet_manager.cc:1428] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
W20260807 18:28:42.986244 3534 tablet.cc:2369] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:28:43.009562 3681 tablet.cc:2369] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:28:43.046653 3834 tablet.cc:2369] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:28:43.180837 3867 raft_consensus.cc:482] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:43.181190 3867 raft_consensus.cc:504] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.182951 3867 leader_election.cc:288] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043), 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867)
I20260807 18:28:43.190147 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.183611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:56967 (local address 127.27.1.193:34867)
0807 18:28:43.183860 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.183866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.183882 (+ 16us) server_negotiation.cc:407] Connection header received
0807 18:28:43.183943 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.183946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.184013 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.184125 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:43.184827 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.185761 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.188250 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.189303 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.189465 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.189563 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:43.189773 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 18:28:43.189906 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:28:43.190140 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.183240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34867 (local address 127.27.1.195:56967)
0807 18:28:43.183530 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:43.183550 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:43.183714 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:43.184154 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:43.184159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:43.184194 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:43.184407 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.184418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.185938 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.185943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.188078 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.188094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.189458 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.189463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.189607 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.189632 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 18:28:43.189715 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 18:28:43.189817 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":2}
I20260807 18:28:43.190872 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.183375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:55709 (local address 127.27.1.194:33043)
0807 18:28:43.183585 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.183593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.184432 (+ 839us) server_negotiation.cc:407] Connection header received
0807 18:28:43.184570 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.184574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.184631 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.184703 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:43.185323 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.186290 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.188637 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.189837 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.190000 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.190068 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:43.190459 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 18:28:43.190576 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 18:28:43.191232 3481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c4ff71ce974982a34af2f2ef7dcaa5" candidate_uuid: "8f90ac215f4742d7adc29c062fadaa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56db8696eb2248fb927c7c02e8c60424" is_pre_election: true
I20260807 18:28:43.191674 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.183377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:33043 (local address 127.27.1.195:55709)
0807 18:28:43.184310 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:43.184328 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:43.184441 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:43.184758 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:43.184761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:43.184868 (+ 107us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:43.185164 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.185172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.186429 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.186434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.188492 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.188503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.190062 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.190067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.190244 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.190292 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 18:28:43.191082 (+ 790us) client_negotiation.cc:240] Negotiation successful
0807 18:28:43.191238 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":125,"thread_start_us":583,"threads_started":1}
I20260807 18:28:43.191974 3481 raft_consensus.cc:2454] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90ac215f4742d7adc29c062fadaa20 in term 0.
I20260807 18:28:43.192696 3706 leader_election.cc:302] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56db8696eb2248fb927c7c02e8c60424, 8f90ac215f4742d7adc29c062fadaa20; no voters:
I20260807 18:28:43.192647 3630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c4ff71ce974982a34af2f2ef7dcaa5" candidate_uuid: "8f90ac215f4742d7adc29c062fadaa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b423204c1fc426e9470b84a0d35d458" is_pre_election: true
I20260807 18:28:43.193143 3867 raft_consensus.cc:2781] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:28:43.193171 3630 raft_consensus.cc:2454] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90ac215f4742d7adc29c062fadaa20 in term 0.
I20260807 18:28:43.193296 3867 raft_consensus.cc:482] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:28:43.193425 3867 raft_consensus.cc:3037] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:43.195807 3867 raft_consensus.cc:504] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.196425 3867 leader_election.cc:288] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [CANDIDATE]: Term 1 election: Requested vote from peers 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043), 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867)
I20260807 18:28:43.196857 3630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c4ff71ce974982a34af2f2ef7dcaa5" candidate_uuid: "8f90ac215f4742d7adc29c062fadaa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b423204c1fc426e9470b84a0d35d458"
I20260807 18:28:43.196900 3481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c4ff71ce974982a34af2f2ef7dcaa5" candidate_uuid: "8f90ac215f4742d7adc29c062fadaa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56db8696eb2248fb927c7c02e8c60424"
I20260807 18:28:43.197055 3630 raft_consensus.cc:3037] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:43.197114 3481 raft_consensus.cc:3037] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:43.199371 3481 raft_consensus.cc:2454] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90ac215f4742d7adc29c062fadaa20 in term 1.
I20260807 18:28:43.199407 3630 raft_consensus.cc:2454] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90ac215f4742d7adc29c062fadaa20 in term 1.
I20260807 18:28:43.199788 3706 leader_election.cc:302] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56db8696eb2248fb927c7c02e8c60424, 8f90ac215f4742d7adc29c062fadaa20; no voters:
I20260807 18:28:43.200100 3867 raft_consensus.cc:2781] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:43.200569 3867 raft_consensus.cc:686] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 LEADER]: Becoming Leader. State: Replica: 8f90ac215f4742d7adc29c062fadaa20, State: Running, Role: LEADER
I20260807 18:28:43.201030 3867 consensus_queue.cc:237] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.205758 3307 catalog_manager.cc:5495] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195). New cstate: current_term: 1 leader_uuid: "8f90ac215f4742d7adc29c062fadaa20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } health_report { overall_health: HEALTHY } } }
I20260807 18:28:43.289968 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.284984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.27.1.195:45635)
0807 18:28:43.285180 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.285184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.285295 (+ 111us) server_negotiation.cc:407] Connection header received
0807 18:28:43.285478 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.285482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.285559 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.285686 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:43.286042 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.286791 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.288113 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.288526 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.288623 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.289004 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:43.289513 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:43.289621 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:43.289701 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 18:28:43.289762 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 18:28:43.297287 3481 raft_consensus.cc:1261] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Refusing update from remote peer 8f90ac215f4742d7adc29c062fadaa20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:43.297331 3630 raft_consensus.cc:1261] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Refusing update from remote peer 8f90ac215f4742d7adc29c062fadaa20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:43.298183 3870 consensus_queue.cc:1035] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:43.298559 3867 consensus_queue.cc:1035] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:43.309815 3873 mvcc.cc:205] Tried to move back new op lower bound from 7315977516215934976 to 7315977515834978304. Current Snapshot: MvccSnapshot[applied={T|T < 7315977516215934976}]
I20260807 18:28:43.312224 3875 mvcc.cc:205] Tried to move back new op lower bound from 7315977516215934976 to 7315977515834978304. Current Snapshot: MvccSnapshot[applied={T|T < 7315977516215934976}]
I20260807 18:28:43.313328 3874 mvcc.cc:205] Tried to move back new op lower bound from 7315977516215934976 to 7315977515834978304. Current Snapshot: MvccSnapshot[applied={T|T < 7315977516215934976}]
I20260807 18:28:43.332252 3307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:43.332564 3307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:43.336001 3307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820: Sending DeleteTablet for 3 replicas of tablet 13c4ff71ce974982a34af2f2ef7dcaa5
I20260807 18:28:43.337150 3608 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c4ff71ce974982a34af2f2ef7dcaa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:43 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:38668
I20260807 18:28:43.337195 3458 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c4ff71ce974982a34af2f2ef7dcaa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:43 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:49898
I20260807 18:28:43.337951 3756 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c4ff71ce974982a34af2f2ef7dcaa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:43 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:60210
I20260807 18:28:43.338721 3307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:28:43.339406 3307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:28:43.346972 3607 tablet_service.cc:1467] Processing CreateTablet for tablet f2de71928daa41768adb1d96d6606cc4 (DEFAULT_TABLE table=test-table [id=30cdfae90c7a4df8a5bb6d0123a66ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:43.347640 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2de71928daa41768adb1d96d6606cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:43.347694 3457 tablet_service.cc:1467] Processing CreateTablet for tablet f2de71928daa41768adb1d96d6606cc4 (DEFAULT_TABLE table=test-table [id=30cdfae90c7a4df8a5bb6d0123a66ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:43.348234 3457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2de71928daa41768adb1d96d6606cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:43.351702 3755 tablet_service.cc:1467] Processing CreateTablet for tablet f2de71928daa41768adb1d96d6606cc4 (DEFAULT_TABLE table=test-table [id=30cdfae90c7a4df8a5bb6d0123a66ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:43.352375 3755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2de71928daa41768adb1d96d6606cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:43.355579 3860 tablet_bootstrap.cc:492] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424: Bootstrap starting.
I20260807 18:28:43.357612 3860 tablet_bootstrap.cc:654] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:43.365911 3861 tablet_bootstrap.cc:492] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20: Bootstrap starting.
I20260807 18:28:43.366060 3859 tablet_bootstrap.cc:492] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458: Bootstrap starting.
I20260807 18:28:43.367939 3861 tablet_bootstrap.cc:654] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:43.368103 3859 tablet_bootstrap.cc:654] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:43.368431 3886 tablet_replica.cc:330] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: stopping tablet replica
I20260807 18:28:43.368930 3886 raft_consensus.cc:2229] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:28:43.369763 3886 raft_consensus.cc:2258] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:43.373811 3860 tablet_bootstrap.cc:492] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424: No bootstrap required, opened a new log
I20260807 18:28:43.373988 3860 ts_tablet_manager.cc:1397] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20260807 18:28:43.375020 3860 raft_consensus.cc:348] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.375321 3860 raft_consensus.cc:374] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:43.375411 3860 raft_consensus.cc:729] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56db8696eb2248fb927c7c02e8c60424, State: Initialized, Role: FOLLOWER
I20260807 18:28:43.375805 3860 consensus_queue.cc:260] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.376592 3860 ts_tablet_manager.cc:1428] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 18:28:43.377651 3882 tablet_replica.cc:330] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: stopping tablet replica
I20260807 18:28:43.378173 3882 raft_consensus.cc:2229] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:43.379258 3885 tablet_replica.cc:330] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: stopping tablet replica
I20260807 18:28:43.379660 3885 raft_consensus.cc:2229] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:43.380235 3885 raft_consensus.cc:2258] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:43.380941 3882 raft_consensus.cc:2258] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:43.382283 3886 ts_tablet_manager.cc:1905] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:43.383164 3859 tablet_bootstrap.cc:492] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458: No bootstrap required, opened a new log
I20260807 18:28:43.383389 3859 ts_tablet_manager.cc:1397] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260807 18:28:43.383471 3861 tablet_bootstrap.cc:492] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20: No bootstrap required, opened a new log
I20260807 18:28:43.383769 3861 ts_tablet_manager.cc:1397] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.011s
I20260807 18:28:43.384435 3885 ts_tablet_manager.cc:1905] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:43.384440 3859 raft_consensus.cc:348] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.384771 3859 raft_consensus.cc:374] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:43.384915 3859 raft_consensus.cc:729] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b423204c1fc426e9470b84a0d35d458, State: Initialized, Role: FOLLOWER
I20260807 18:28:43.384711 3861 raft_consensus.cc:348] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.385129 3861 raft_consensus.cc:374] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:43.385313 3861 raft_consensus.cc:729] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90ac215f4742d7adc29c062fadaa20, State: Initialized, Role: FOLLOWER
I20260807 18:28:43.385382 3859 consensus_queue.cc:260] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.386032 3859 ts_tablet_manager.cc:1428] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 18:28:43.385823 3861 consensus_queue.cc:260] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.386508 3861 ts_tablet_manager.cc:1428] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260807 18:28:43.383214 3882 ts_tablet_manager.cc:1905] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:43.394672 3886 ts_tablet_manager.cc:1918] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:43.394865 3886 log.cc:1196] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/13c4ff71ce974982a34af2f2ef7dcaa5
I20260807 18:28:43.395303 3882 ts_tablet_manager.cc:1918] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:43.395488 3886 ts_tablet_manager.cc:1939] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 8f90ac215f4742d7adc29c062fadaa20: Deleting consensus metadata
I20260807 18:28:43.395495 3882 log.cc:1196] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/13c4ff71ce974982a34af2f2ef7dcaa5
I20260807 18:28:43.396201 3882 ts_tablet_manager.cc:1939] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 5b423204c1fc426e9470b84a0d35d458: Deleting consensus metadata
I20260807 18:28:43.397367 3291 catalog_manager.cc:4841] TS 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635): tablet 13c4ff71ce974982a34af2f2ef7dcaa5 (table test-table [id=0d717db9e5fe48a1b7d3173d5bfc7f63]) successfully deleted
I20260807 18:28:43.397898 3290 catalog_manager.cc:4841] TS 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043): tablet 13c4ff71ce974982a34af2f2ef7dcaa5 (table test-table [id=0d717db9e5fe48a1b7d3173d5bfc7f63]) successfully deleted
I20260807 18:28:43.400978 3885 ts_tablet_manager.cc:1918] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:43.401136 3885 log.cc:1196] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/13c4ff71ce974982a34af2f2ef7dcaa5
I20260807 18:28:43.401666 3885 ts_tablet_manager.cc:1939] T 13c4ff71ce974982a34af2f2ef7dcaa5 P 56db8696eb2248fb927c7c02e8c60424: Deleting consensus metadata
I20260807 18:28:43.403047 3289 catalog_manager.cc:4841] TS 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867): tablet 13c4ff71ce974982a34af2f2ef7dcaa5 (table test-table [id=0d717db9e5fe48a1b7d3173d5bfc7f63]) successfully deleted
I20260807 18:28:43.419853 3866 raft_consensus.cc:482] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:43.420096 3866 raft_consensus.cc:504] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.421885 3866 leader_election.cc:288] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867), 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635)
I20260807 18:28:43.429666 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.422753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:58853 (local address 127.27.1.195:45635)
0807 18:28:43.422958 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.422966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.423608 (+ 642us) server_negotiation.cc:407] Connection header received
0807 18:28:43.423760 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.423766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.423854 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.423955 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:43.424614 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.425392 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.427366 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.428460 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.428618 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.428700 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:43.429239 (+ 539us) server_negotiation.cc:299] Negotiation successful
0807 18:28:43.429395 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 18:28:43.430400 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.422186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:45635 (local address 127.27.1.194:58853)
0807 18:28:43.423492 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:43.423510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:43.423621 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:43.423989 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:43.423993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:43.424023 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:43.424462 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.424476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.425523 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.425531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.427195 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.427216 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.428852 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.428858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.429043 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.429075 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 18:28:43.429970 (+ 895us) client_negotiation.cc:240] Negotiation successful
0807 18:28:43.430070 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"mutex_wait_us":3,"thread_start_us":956,"threads_started":1}
I20260807 18:28:43.430769 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.422417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:44257 (local address 127.27.1.193:34867)
0807 18:28:43.422612 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.422619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.423014 (+ 395us) server_negotiation.cc:407] Connection header received
0807 18:28:43.423403 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.423409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.423507 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.423576 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:43.424492 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.425625 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.428256 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.429531 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.429673 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.429744 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:43.430304 (+ 560us) server_negotiation.cc:299] Negotiation successful
0807 18:28:43.430467 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 18:28:43.430790 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.422030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34867 (local address 127.27.1.194:44257)
0807 18:28:43.422914 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:43.422934 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:43.423201 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:43.423805 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:43.423808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:43.423829 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:43.424266 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.424285 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.425797 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.425803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.428065 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.428083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.429932 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.429938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.430124 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.430161 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 18:28:43.430254 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 18:28:43.430406 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":1,"thread_start_us":613,"threads_started":1}
I20260807 18:28:43.431414 3784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2de71928daa41768adb1d96d6606cc4" candidate_uuid: "5b423204c1fc426e9470b84a0d35d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90ac215f4742d7adc29c062fadaa20" is_pre_election: true
I20260807 18:28:43.431550 3481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2de71928daa41768adb1d96d6606cc4" candidate_uuid: "5b423204c1fc426e9470b84a0d35d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56db8696eb2248fb927c7c02e8c60424" is_pre_election: true
I20260807 18:28:43.431797 3481 raft_consensus.cc:2454] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b423204c1fc426e9470b84a0d35d458 in term 0.
I20260807 18:28:43.431914 3784 raft_consensus.cc:2454] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b423204c1fc426e9470b84a0d35d458 in term 0.
I20260807 18:28:43.432294 3554 leader_election.cc:302] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56db8696eb2248fb927c7c02e8c60424, 5b423204c1fc426e9470b84a0d35d458; no voters:
I20260807 18:28:43.432708 3866 raft_consensus.cc:2781] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:28:43.432842 3866 raft_consensus.cc:482] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:28:43.432957 3866 raft_consensus.cc:3037] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:43.435626 3866 raft_consensus.cc:504] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.436323 3866 leader_election.cc:288] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [CANDIDATE]: Term 1 election: Requested vote from peers 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867), 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635)
I20260807 18:28:43.436849 3784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2de71928daa41768adb1d96d6606cc4" candidate_uuid: "5b423204c1fc426e9470b84a0d35d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90ac215f4742d7adc29c062fadaa20"
I20260807 18:28:43.436810 3481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2de71928daa41768adb1d96d6606cc4" candidate_uuid: "5b423204c1fc426e9470b84a0d35d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56db8696eb2248fb927c7c02e8c60424"
I20260807 18:28:43.437063 3481 raft_consensus.cc:3037] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:43.437093 3784 raft_consensus.cc:3037] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:43.440135 3481 raft_consensus.cc:2454] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b423204c1fc426e9470b84a0d35d458 in term 1.
I20260807 18:28:43.440387 3784 raft_consensus.cc:2454] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b423204c1fc426e9470b84a0d35d458 in term 1.
I20260807 18:28:43.440538 3554 leader_election.cc:302] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56db8696eb2248fb927c7c02e8c60424, 5b423204c1fc426e9470b84a0d35d458; no voters:
I20260807 18:28:43.440842 3866 raft_consensus.cc:2781] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:43.441349 3866 raft_consensus.cc:686] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 1 LEADER]: Becoming Leader. State: Replica: 5b423204c1fc426e9470b84a0d35d458, State: Running, Role: LEADER
I20260807 18:28:43.441956 3866 consensus_queue.cc:237] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.445505 3308 catalog_manager.cc:5495] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194). New cstate: current_term: 1 leader_uuid: "5b423204c1fc426e9470b84a0d35d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } health_report { overall_health: UNKNOWN } } }
Aug 07 18:28:43 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127320, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.254@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:43.487839 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.473519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.27.1.254:43315)
0807 18:28:43.474252 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.474263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.474502 (+ 239us) server_negotiation.cc:407] Connection header received
0807 18:28:43.474701 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.474712 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.474771 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.474890 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:43.476027 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.476762 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.477948 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.478368 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.478493 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.482085 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:43.482089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:43.482093 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:43.482121 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:43.484917 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:43.485605 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:43.485617 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:43.485620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:43.485730 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:43.486095 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:43.486101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:43.486104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:43.486409 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:43.486583 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:43.487102 (+ 519us) server_negotiation.cc:299] Negotiation successful
0807 18:28:43.487509 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":539,"thread_start_us":333,"threads_started":1}
I20260807 18:28:43.487891 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.473550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:43315 (local address 127.0.0.1:48098)
0807 18:28:43.474396 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:43.474416 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:43.474585 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:43.474906 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:43.474910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:43.475679 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:43.475873 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.475883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.476893 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.476901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.477809 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.477820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.478531 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.478537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.478705 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.479493 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:43.479510 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:43.481887 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:43.485052 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:43.485066 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:43.485074 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:43.485479 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:43.485839 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:43.485844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:43.485848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:43.485990 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:43.486581 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:43.486591 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:43.486798 (+ 207us) client_negotiation.cc:769] Sending connection context
0807 18:28:43.486998 (+ 200us) client_negotiation.cc:240] Negotiation successful
0807 18:28:43.487334 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":567,"threads_started":1}
I20260807 18:28:43.493686 3308 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:48098:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 18:28:43.494987 3308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:28:43.502120 3608 tablet_service.cc:1467] Processing CreateTablet for tablet 6729bce26e694e388c1755ebac6f9145 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4f2ec0061d44016a153b1ad09053ac4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 18:28:43.502792 3608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6729bce26e694e388c1755ebac6f9145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:43.502703 3458 tablet_service.cc:1467] Processing CreateTablet for tablet 6729bce26e694e388c1755ebac6f9145 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4f2ec0061d44016a153b1ad09053ac4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 18:28:43.502734 3756 tablet_service.cc:1467] Processing CreateTablet for tablet 6729bce26e694e388c1755ebac6f9145 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4f2ec0061d44016a153b1ad09053ac4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 18:28:43.503299 3458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6729bce26e694e388c1755ebac6f9145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:43.503398 3756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6729bce26e694e388c1755ebac6f9145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:43.510164 3859 tablet_bootstrap.cc:492] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: Bootstrap starting.
I20260807 18:28:43.510385 3860 tablet_bootstrap.cc:492] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: Bootstrap starting.
I20260807 18:28:43.511746 3859 tablet_bootstrap.cc:654] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:43.512068 3860 tablet_bootstrap.cc:654] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:43.513082 3861 tablet_bootstrap.cc:492] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: Bootstrap starting.
I20260807 18:28:43.514320 3859 tablet_bootstrap.cc:492] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: No bootstrap required, opened a new log
I20260807 18:28:43.514496 3859 ts_tablet_manager.cc:1397] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260807 18:28:43.514780 3861 tablet_bootstrap.cc:654] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:43.515290 3859 raft_consensus.cc:348] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } }
I20260807 18:28:43.515533 3859 raft_consensus.cc:374] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:43.515623 3859 raft_consensus.cc:729] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b423204c1fc426e9470b84a0d35d458, State: Initialized, Role: FOLLOWER
I20260807 18:28:43.515899 3859 consensus_queue.cc:260] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } }
I20260807 18:28:43.516526 3859 ts_tablet_manager.cc:1428] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260807 18:28:43.516422 3866 tablet_replica.cc:439] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } }
I20260807 18:28:43.516762 3866 tablet_replica.cc:442] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:28:43.519277 3860 tablet_bootstrap.cc:492] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: No bootstrap required, opened a new log
I20260807 18:28:43.519449 3860 ts_tablet_manager.cc:1397] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260807 18:28:43.520071 3860 raft_consensus.cc:348] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } }
I20260807 18:28:43.520237 3860 raft_consensus.cc:374] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:43.520283 3860 raft_consensus.cc:729] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56db8696eb2248fb927c7c02e8c60424, State: Initialized, Role: FOLLOWER
I20260807 18:28:43.520506 3860 consensus_queue.cc:260] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } }
I20260807 18:28:43.521104 3860 ts_tablet_manager.cc:1428] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260807 18:28:43.520927 3868 tablet_replica.cc:439] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } }
I20260807 18:28:43.521312 3868 tablet_replica.cc:442] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:28:43.529307 3861 tablet_bootstrap.cc:492] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: No bootstrap required, opened a new log
I20260807 18:28:43.529496 3861 ts_tablet_manager.cc:1397] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.013s
I20260807 18:28:43.529654 3866 raft_consensus.cc:482] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:43.529829 3866 raft_consensus.cc:504] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } }
I20260807 18:28:43.530258 3861 raft_consensus.cc:348] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } }
I20260807 18:28:43.530531 3866 leader_election.cc:288] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635), 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867)
I20260807 18:28:43.530514 3861 raft_consensus.cc:374] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:43.530893 3861 raft_consensus.cc:729] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90ac215f4742d7adc29c062fadaa20, State: Initialized, Role: FOLLOWER
I20260807 18:28:43.531008 3784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6729bce26e694e388c1755ebac6f9145" candidate_uuid: "5b423204c1fc426e9470b84a0d35d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90ac215f4742d7adc29c062fadaa20" is_pre_election: true
I20260807 18:28:43.531179 3481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6729bce26e694e388c1755ebac6f9145" candidate_uuid: "5b423204c1fc426e9470b84a0d35d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56db8696eb2248fb927c7c02e8c60424" is_pre_election: true
I20260807 18:28:43.531231 3861 consensus_queue.cc:260] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } }
I20260807 18:28:43.531389 3481 raft_consensus.cc:2454] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b423204c1fc426e9470b84a0d35d458 in term 0.
I20260807 18:28:43.531560 3784 raft_consensus.cc:2454] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b423204c1fc426e9470b84a0d35d458 in term 0.
I20260807 18:28:43.531806 3554 leader_election.cc:302] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56db8696eb2248fb927c7c02e8c60424, 5b423204c1fc426e9470b84a0d35d458; no voters:
I20260807 18:28:43.532094 3866 raft_consensus.cc:2781] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:28:43.532204 3866 raft_consensus.cc:482] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:28:43.532325 3866 raft_consensus.cc:3037] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:43.532497 3870 tablet_replica.cc:439] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } }
I20260807 18:28:43.532831 3870 tablet_replica.cc:442] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:28:43.534615 3866 raft_consensus.cc:504] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } }
I20260807 18:28:43.535331 3866 leader_election.cc:288] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635), 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867)
I20260807 18:28:43.535527 3861 ts_tablet_manager.cc:1428] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260807 18:28:43.535666 3784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6729bce26e694e388c1755ebac6f9145" candidate_uuid: "5b423204c1fc426e9470b84a0d35d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90ac215f4742d7adc29c062fadaa20"
I20260807 18:28:43.535876 3784 raft_consensus.cc:3037] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:43.536062 3481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6729bce26e694e388c1755ebac6f9145" candidate_uuid: "5b423204c1fc426e9470b84a0d35d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56db8696eb2248fb927c7c02e8c60424"
I20260807 18:28:43.536281 3481 raft_consensus.cc:3037] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:43.538094 3784 raft_consensus.cc:2454] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b423204c1fc426e9470b84a0d35d458 in term 1.
I20260807 18:28:43.539175 3556 leader_election.cc:302] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b423204c1fc426e9470b84a0d35d458, 8f90ac215f4742d7adc29c062fadaa20; no voters:
I20260807 18:28:43.539259 3481 raft_consensus.cc:2454] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b423204c1fc426e9470b84a0d35d458 in term 1.
I20260807 18:28:43.539463 3866 raft_consensus.cc:2781] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:43.539661 3866 raft_consensus.cc:686] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [term 1 LEADER]: Becoming Leader. State: Replica: 5b423204c1fc426e9470b84a0d35d458, State: Running, Role: LEADER
I20260807 18:28:43.539933 3866 consensus_queue.cc:237] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } }
I20260807 18:28:43.541911 3892 tablet_replica.cc:439] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: TxnStatusTablet state changed. Reason: New leader 5b423204c1fc426e9470b84a0d35d458. Latest consensus state: current_term: 1 leader_uuid: "5b423204c1fc426e9470b84a0d35d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } }
I20260807 18:28:43.542318 3892 tablet_replica.cc:442] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: This TxnStatusTablet replica's current role is: LEADER
I20260807 18:28:43.543071 3905 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 18:28:43.543356 3905 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 18:28:43.543496 3308 catalog_manager.cc:5495] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194). New cstate: current_term: 1 leader_uuid: "5b423204c1fc426e9470b84a0d35d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } health_report { overall_health: UNKNOWN } } }
I20260807 18:28:43.563738 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.555718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.27.1.194:33043)
0807 18:28:43.556513 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.556521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.556539 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:28:43.556631 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.556636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.556753 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.556916 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:43.557497 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.558763 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.560816 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.561576 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.561697 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.562278 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:43.562996 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:43.563166 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:43.563316 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 18:28:43.563410 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665}
I20260807 18:28:43.563884 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.555682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:33043 (local address 127.0.0.1:38684)
0807 18:28:43.556068 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:43.556090 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:43.556222 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:43.557098 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:43.557103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:43.557116 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 18:28:43.557314 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.557326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.558935 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.558941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.560658 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.560674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.561766 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.561773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.561993 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.562077 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 18:28:43.563118 (+ 1041us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 18:28:43.563173 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 18:28:43.563272 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 18:28:43.563474 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260807 18:28:43.568477 3481 raft_consensus.cc:1261] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Refusing update from remote peer 5b423204c1fc426e9470b84a0d35d458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:43.568499 3784 raft_consensus.cc:1261] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Refusing update from remote peer 5b423204c1fc426e9470b84a0d35d458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:43.569466 3892 consensus_queue.cc:1035] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:43.569844 3866 consensus_queue.cc:1035] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:43.582999 3868 tablet_replica.cc:439] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: TxnStatusTablet state changed. Reason: New leader 5b423204c1fc426e9470b84a0d35d458. Latest consensus state: current_term: 1 leader_uuid: "5b423204c1fc426e9470b84a0d35d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } }
I20260807 18:28:43.583344 3868 tablet_replica.cc:442] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:28:43.588608 3892 tablet_replica.cc:439] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b423204c1fc426e9470b84a0d35d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } }
I20260807 18:28:43.588896 3892 tablet_replica.cc:442] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: This TxnStatusTablet replica's current role is: LEADER
I20260807 18:28:43.593508 3868 tablet_replica.cc:439] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b423204c1fc426e9470b84a0d35d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } }
I20260807 18:28:43.593741 3868 tablet_replica.cc:442] T 6729bce26e694e388c1755ebac6f9145 P 56db8696eb2248fb927c7c02e8c60424: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:28:43.591617 3870 tablet_replica.cc:439] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: TxnStatusTablet state changed. Reason: New leader 5b423204c1fc426e9470b84a0d35d458. Latest consensus state: current_term: 1 leader_uuid: "5b423204c1fc426e9470b84a0d35d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } }
I20260807 18:28:43.596427 3870 tablet_replica.cc:442] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:28:43.598361 3866 tablet_replica.cc:439] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b423204c1fc426e9470b84a0d35d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } }
I20260807 18:28:43.598663 3866 tablet_replica.cc:442] T 6729bce26e694e388c1755ebac6f9145 P 5b423204c1fc426e9470b84a0d35d458: This TxnStatusTablet replica's current role is: LEADER
I20260807 18:28:43.603188 3870 tablet_replica.cc:439] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b423204c1fc426e9470b84a0d35d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } } peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } }
I20260807 18:28:43.603451 3870 tablet_replica.cc:442] T 6729bce26e694e388c1755ebac6f9145 P 8f90ac215f4742d7adc29c062fadaa20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 18:28:43.605754 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.597761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.27.1.194:33043)
0807 18:28:43.598088 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.598094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.598112 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:28:43.598182 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.598187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.598294 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.598414 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:43.599085 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.600326 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.602549 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.603286 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.603417 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.604711 (+ 1294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:43.604844 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:43.604935 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:43.605383 (+ 448us) server_negotiation.cc:299] Negotiation successful
0807 18:28:43.605465 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260807 18:28:43.629240 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.618028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:48723 (local address 127.27.1.194:33043)
0807 18:28:43.618221 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.618227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.622351 (+ 4124us) server_negotiation.cc:407] Connection header received
0807 18:28:43.622587 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.622593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.622676 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.622776 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:43.623443 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.624624 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.626834 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.627554 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.627683 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.628287 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:43.628441 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:43.628531 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:43.628715 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 18:28:43.628878 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":63}
I20260807 18:28:43.629396 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.618047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:33043 (local address 127.27.1.194:48723)
0807 18:28:43.622228 (+ 4181us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:43.622249 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:43.622443 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:43.622822 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:43.622826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:43.622839 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 18:28:43.623274 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.623287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.624798 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.624804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.626663 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:43.626679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.627779 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.627786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:43.628005 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.628077 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 18:28:43.628558 (+ 481us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 18:28:43.628577 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 18:28:43.628680 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 18:28:43.628884 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3993,"mutex_wait_us":1,"thread_start_us":3819,"threads_started":1}
I20260807 18:28:43.642457 3481 raft_consensus.cc:1261] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Refusing update from remote peer 5b423204c1fc426e9470b84a0d35d458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:43.642508 3784 raft_consensus.cc:1261] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Refusing update from remote peer 5b423204c1fc426e9470b84a0d35d458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:43.643087 3892 consensus_queue.cc:1035] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:43.643476 3866 consensus_queue.cc:1035] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:43.762764 3308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:43.763038 3308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:43.766835 3308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820: Sending DeleteTablet for 3 replicas of tablet f2de71928daa41768adb1d96d6606cc4
I20260807 18:28:43.767895 3608 tablet_service.cc:1514] Processing DeleteTablet for tablet f2de71928daa41768adb1d96d6606cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:43 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:38668
I20260807 18:28:43.768057 3756 tablet_service.cc:1514] Processing DeleteTablet for tablet f2de71928daa41768adb1d96d6606cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:43 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:60210
I20260807 18:28:43.768162 3882 tablet_replica.cc:330] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458: stopping tablet replica
I20260807 18:28:43.768366 3886 tablet_replica.cc:330] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20: stopping tablet replica
I20260807 18:28:43.768368 3882 raft_consensus.cc:2229] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:28:43.768569 3886 raft_consensus.cc:2229] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:43.768813 3886 raft_consensus.cc:2258] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:43.768863 3882 raft_consensus.cc:2258] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:43.769737 3458 tablet_service.cc:1514] Processing DeleteTablet for tablet f2de71928daa41768adb1d96d6606cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:43 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:49898
I20260807 18:28:43.770054 3885 tablet_replica.cc:330] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424: stopping tablet replica
I20260807 18:28:43.770051 3882 ts_tablet_manager.cc:1905] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:43.770242 3885 raft_consensus.cc:2229] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:43.770445 3885 raft_consensus.cc:2258] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:43.770689 3886 ts_tablet_manager.cc:1905] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:43.772270 3885 ts_tablet_manager.cc:1905] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:43.777084 3882 ts_tablet_manager.cc:1918] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:28:43.777253 3882 log.cc:1196] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/f2de71928daa41768adb1d96d6606cc4
I20260807 18:28:43.777700 3882 ts_tablet_manager.cc:1939] T f2de71928daa41768adb1d96d6606cc4 P 5b423204c1fc426e9470b84a0d35d458: Deleting consensus metadata
I20260807 18:28:43.778277 3886 ts_tablet_manager.cc:1918] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:28:43.778467 3886 log.cc:1196] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/f2de71928daa41768adb1d96d6606cc4
I20260807 18:28:43.778925 3886 ts_tablet_manager.cc:1939] T f2de71928daa41768adb1d96d6606cc4 P 8f90ac215f4742d7adc29c062fadaa20: Deleting consensus metadata
I20260807 18:28:43.778906 3290 catalog_manager.cc:4841] TS 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043): tablet f2de71928daa41768adb1d96d6606cc4 (table test-table [id=30cdfae90c7a4df8a5bb6d0123a66ca9]) successfully deleted
I20260807 18:28:43.779733 3885 ts_tablet_manager.cc:1918] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:28:43.779969 3885 log.cc:1196] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/f2de71928daa41768adb1d96d6606cc4
I20260807 18:28:43.780440 3885 ts_tablet_manager.cc:1939] T f2de71928daa41768adb1d96d6606cc4 P 56db8696eb2248fb927c7c02e8c60424: Deleting consensus metadata
I20260807 18:28:43.781576 3289 catalog_manager.cc:4841] TS 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867): tablet f2de71928daa41768adb1d96d6606cc4 (table test-table [id=30cdfae90c7a4df8a5bb6d0123a66ca9]) successfully deleted
I20260807 18:28:43.796569 3291 catalog_manager.cc:4841] TS 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635): tablet f2de71928daa41768adb1d96d6606cc4 (table test-table [id=30cdfae90c7a4df8a5bb6d0123a66ca9]) successfully deleted
Aug 07 18:28:43 dist-test-slave-1xrh krb5kdc[3259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 18:28:43 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:43 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
W20260807 18:28:43.886382 3899 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:43.875730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.27.1.254:43315)
0807 18:28:43.875965 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.875975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.876551 (+ 576us) server_negotiation.cc:407] Connection header received
0807 18:28:43.876775 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.876781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.876873 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.876987 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:43.878340 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.879089 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.880361 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.880819 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.880968 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.886075 (+ 5107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48102: BlockingRecv error: recv got EOF from 127.0.0.1:48102 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 18:28:43 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM
I20260807 18:28:43.909253 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:43.893611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.27.1.254:43315)
0807 18:28:43.893795 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:43.893799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:43.895698 (+ 1899us) server_negotiation.cc:407] Connection header received
0807 18:28:43.895908 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:43.895912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:43.895974 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:43.896099 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:43.897495 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.898193 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.899522 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:43.899946 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:43.900088 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:43.903948 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:43.903953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:43.903956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:43.903992 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:43.906552 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:43.907199 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:43.907206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:43.907209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:43.907346 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:43.907726 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:43.907733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:43.907740 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:43.908020 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:43.908289 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:43.908602 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 18:28:43.908878 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 18:28:43.912797 3308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:28:43.914006 3308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:28:43.922658 3458 tablet_service.cc:1467] Processing CreateTablet for tablet b3bd90979e244f258a43f3d066868f98 (DEFAULT_TABLE table=test-table [id=12cda6359240433eb4f90f02a15e4a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:43.923192 3458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bd90979e244f258a43f3d066868f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:43.923817 3608 tablet_service.cc:1467] Processing CreateTablet for tablet b3bd90979e244f258a43f3d066868f98 (DEFAULT_TABLE table=test-table [id=12cda6359240433eb4f90f02a15e4a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:43.924255 3608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bd90979e244f258a43f3d066868f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:43.929602 3860 tablet_bootstrap.cc:492] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424: Bootstrap starting.
I20260807 18:28:43.923105 3756 tablet_service.cc:1467] Processing CreateTablet for tablet b3bd90979e244f258a43f3d066868f98 (DEFAULT_TABLE table=test-table [id=12cda6359240433eb4f90f02a15e4a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:43.930044 3756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bd90979e244f258a43f3d066868f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:43.931380 3860 tablet_bootstrap.cc:654] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:43.938977 3861 tablet_bootstrap.cc:492] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20: Bootstrap starting.
I20260807 18:28:43.940744 3861 tablet_bootstrap.cc:654] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:43.941556 3859 tablet_bootstrap.cc:492] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458: Bootstrap starting.
I20260807 18:28:43.943442 3859 tablet_bootstrap.cc:654] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:43.952822 3859 tablet_bootstrap.cc:492] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458: No bootstrap required, opened a new log
I20260807 18:28:43.953107 3859 ts_tablet_manager.cc:1397] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260807 18:28:43.954172 3859 raft_consensus.cc:348] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.954725 3859 raft_consensus.cc:374] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:43.954898 3859 raft_consensus.cc:729] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b423204c1fc426e9470b84a0d35d458, State: Initialized, Role: FOLLOWER
I20260807 18:28:43.955330 3859 consensus_queue.cc:260] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.959019 3859 ts_tablet_manager.cc:1428] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260807 18:28:43.960321 3861 tablet_bootstrap.cc:492] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20: No bootstrap required, opened a new log
I20260807 18:28:43.960496 3861 ts_tablet_manager.cc:1397] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.004s
I20260807 18:28:43.961308 3861 raft_consensus.cc:348] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.961570 3861 raft_consensus.cc:374] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:43.961668 3861 raft_consensus.cc:729] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90ac215f4742d7adc29c062fadaa20, State: Initialized, Role: FOLLOWER
I20260807 18:28:43.961900 3861 consensus_queue.cc:260] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.962476 3861 ts_tablet_manager.cc:1428] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 18:28:43.964258 3860 tablet_bootstrap.cc:492] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424: No bootstrap required, opened a new log
I20260807 18:28:43.964459 3860 ts_tablet_manager.cc:1397] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.012s
I20260807 18:28:43.965229 3860 raft_consensus.cc:348] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.965517 3860 raft_consensus.cc:374] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:43.965628 3860 raft_consensus.cc:729] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56db8696eb2248fb927c7c02e8c60424, State: Initialized, Role: FOLLOWER
I20260807 18:28:43.965919 3860 consensus_queue.cc:260] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:43.966645 3860 ts_tablet_manager.cc:1428] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260807 18:28:44.251111 3870 raft_consensus.cc:482] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:44.251348 3870 raft_consensus.cc:504] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.252074 3870 leader_election.cc:288] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867), 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043)
I20260807 18:28:44.252753 3629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bd90979e244f258a43f3d066868f98" candidate_uuid: "8f90ac215f4742d7adc29c062fadaa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b423204c1fc426e9470b84a0d35d458" is_pre_election: true
I20260807 18:28:44.252749 3481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bd90979e244f258a43f3d066868f98" candidate_uuid: "8f90ac215f4742d7adc29c062fadaa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56db8696eb2248fb927c7c02e8c60424" is_pre_election: true
I20260807 18:28:44.253041 3629 raft_consensus.cc:2454] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90ac215f4742d7adc29c062fadaa20 in term 0.
I20260807 18:28:44.253055 3481 raft_consensus.cc:2454] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90ac215f4742d7adc29c062fadaa20 in term 0.
I20260807 18:28:44.253579 3706 leader_election.cc:302] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56db8696eb2248fb927c7c02e8c60424, 8f90ac215f4742d7adc29c062fadaa20; no voters:
I20260807 18:28:44.254220 3870 raft_consensus.cc:2781] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:28:44.254345 3870 raft_consensus.cc:482] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:28:44.254462 3870 raft_consensus.cc:3037] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:44.257416 3870 raft_consensus.cc:504] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.258379 3870 leader_election.cc:288] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [CANDIDATE]: Term 1 election: Requested vote from peers 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867), 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043)
I20260807 18:28:44.258620 3629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bd90979e244f258a43f3d066868f98" candidate_uuid: "8f90ac215f4742d7adc29c062fadaa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b423204c1fc426e9470b84a0d35d458"
I20260807 18:28:44.258836 3629 raft_consensus.cc:3037] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:44.259560 3481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bd90979e244f258a43f3d066868f98" candidate_uuid: "8f90ac215f4742d7adc29c062fadaa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56db8696eb2248fb927c7c02e8c60424"
I20260807 18:28:44.259785 3481 raft_consensus.cc:3037] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:44.262532 3629 raft_consensus.cc:2454] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90ac215f4742d7adc29c062fadaa20 in term 1.
I20260807 18:28:44.262686 3481 raft_consensus.cc:2454] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90ac215f4742d7adc29c062fadaa20 in term 1.
I20260807 18:28:44.263001 3707 leader_election.cc:302] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b423204c1fc426e9470b84a0d35d458, 8f90ac215f4742d7adc29c062fadaa20; no voters:
I20260807 18:28:44.263463 3870 raft_consensus.cc:2781] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:44.267411 3870 raft_consensus.cc:686] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 LEADER]: Becoming Leader. State: Replica: 8f90ac215f4742d7adc29c062fadaa20, State: Running, Role: LEADER
I20260807 18:28:44.267688 3870 consensus_queue.cc:237] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.273047 3308 catalog_manager.cc:5495] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195). New cstate: current_term: 1 leader_uuid: "8f90ac215f4742d7adc29c062fadaa20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } health_report { overall_health: HEALTHY } } }
I20260807 18:28:44.322510 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.315286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.27.1.195:45635)
0807 18:28:44.316280 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:44.316284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:44.316300 (+ 16us) server_negotiation.cc:407] Connection header received
0807 18:28:44.316356 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:44.316359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:44.316420 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:44.316502 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:44.317041 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.318071 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.320314 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.321054 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.321161 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.321742 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:44.321842 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:44.321939 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:44.322098 (+ 159us) server_negotiation.cc:299] Negotiation successful
0807 18:28:44.322162 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":800,"threads_started":1}
I20260807 18:28:44.325670 3629 raft_consensus.cc:1261] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Refusing update from remote peer 8f90ac215f4742d7adc29c062fadaa20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:44.325953 3481 raft_consensus.cc:1261] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Refusing update from remote peer 8f90ac215f4742d7adc29c062fadaa20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:44.326509 3870 consensus_queue.cc:1035] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:44.326900 3938 consensus_queue.cc:1035] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:44.347805 3305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:44.348129 3305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:44.352028 3305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820: Sending DeleteTablet for 3 replicas of tablet b3bd90979e244f258a43f3d066868f98
I20260807 18:28:44.353204 3458 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bd90979e244f258a43f3d066868f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:44 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:49898
I20260807 18:28:44.353659 3756 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bd90979e244f258a43f3d066868f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:44 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:60210
I20260807 18:28:44.354179 3608 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bd90979e244f258a43f3d066868f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:44 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:38668
I20260807 18:28:44.354630 3607 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bd90979e244f258a43f3d066868f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:44 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:38668
I20260807 18:28:44.354558 3308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:28:44.355331 3308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:28:44.361853 3953 tablet_replica.cc:330] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20: stopping tablet replica
I20260807 18:28:44.362139 3953 raft_consensus.cc:2229] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:28:44.362674 3953 raft_consensus.cc:2258] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:44.364427 3457 tablet_service.cc:1467] Processing CreateTablet for tablet b6479e3003394bdb8ff68562d61ec9dd (DEFAULT_TABLE table=test-table [id=3689de244dd84f4fbe14512241f6e5bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:44.364521 3607 tablet_service.cc:1467] Processing CreateTablet for tablet b6479e3003394bdb8ff68562d61ec9dd (DEFAULT_TABLE table=test-table [id=3689de244dd84f4fbe14512241f6e5bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:44.364918 3457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6479e3003394bdb8ff68562d61ec9dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:44.364943 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6479e3003394bdb8ff68562d61ec9dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:44.371925 3756 tablet_service.cc:1467] Processing CreateTablet for tablet b6479e3003394bdb8ff68562d61ec9dd (DEFAULT_TABLE table=test-table [id=3689de244dd84f4fbe14512241f6e5bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:28:44.372488 3756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6479e3003394bdb8ff68562d61ec9dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:44.373746 3860 tablet_bootstrap.cc:492] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424: Bootstrap starting.
I20260807 18:28:44.374011 3953 ts_tablet_manager.cc:1905] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:44.375445 3860 tablet_bootstrap.cc:654] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:44.380702 3953 ts_tablet_manager.cc:1918] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:44.380856 3953 log.cc:1196] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/b3bd90979e244f258a43f3d066868f98
I20260807 18:28:44.381376 3953 ts_tablet_manager.cc:1939] T b3bd90979e244f258a43f3d066868f98 P 8f90ac215f4742d7adc29c062fadaa20: Deleting consensus metadata
I20260807 18:28:44.382524 3291 catalog_manager.cc:4841] TS 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635): tablet b3bd90979e244f258a43f3d066868f98 (table test-table [id=12cda6359240433eb4f90f02a15e4a2c]) successfully deleted
I20260807 18:28:44.383358 3861 tablet_bootstrap.cc:492] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20: Bootstrap starting.
I20260807 18:28:44.383415 3859 tablet_bootstrap.cc:492] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458: Bootstrap starting.
I20260807 18:28:44.384290 3954 tablet_replica.cc:330] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458: stopping tablet replica
I20260807 18:28:44.384531 3954 raft_consensus.cc:2229] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:44.384845 3954 raft_consensus.cc:2258] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:44.385238 3861 tablet_bootstrap.cc:654] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:44.385318 3952 tablet_replica.cc:330] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424: stopping tablet replica
I20260807 18:28:44.385462 3859 tablet_bootstrap.cc:654] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:44.385520 3952 raft_consensus.cc:2229] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:44.385773 3952 raft_consensus.cc:2258] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:44.387673 3860 tablet_bootstrap.cc:492] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424: No bootstrap required, opened a new log
I20260807 18:28:44.387844 3860 ts_tablet_manager.cc:1397] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260807 18:28:44.388495 3860 raft_consensus.cc:348] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.388713 3860 raft_consensus.cc:374] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:44.388792 3860 raft_consensus.cc:729] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56db8696eb2248fb927c7c02e8c60424, State: Initialized, Role: FOLLOWER
I20260807 18:28:44.388811 3952 ts_tablet_manager.cc:1905] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:44.389045 3860 consensus_queue.cc:260] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.390568 3954 ts_tablet_manager.cc:1905] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:44.392066 3860 ts_tablet_manager.cc:1428] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260807 18:28:44.394755 3861 tablet_bootstrap.cc:492] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20: No bootstrap required, opened a new log
I20260807 18:28:44.394995 3861 ts_tablet_manager.cc:1397] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20260807 18:28:44.395980 3861 raft_consensus.cc:348] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.396317 3861 raft_consensus.cc:374] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:44.396512 3861 raft_consensus.cc:729] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90ac215f4742d7adc29c062fadaa20, State: Initialized, Role: FOLLOWER
I20260807 18:28:44.396857 3861 consensus_queue.cc:260] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.397459 3861 ts_tablet_manager.cc:1428] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 18:28:44.397668 3952 ts_tablet_manager.cc:1918] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:44.397826 3952 log.cc:1196] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/b3bd90979e244f258a43f3d066868f98
I20260807 18:28:44.398377 3952 ts_tablet_manager.cc:1939] T b3bd90979e244f258a43f3d066868f98 P 56db8696eb2248fb927c7c02e8c60424: Deleting consensus metadata
I20260807 18:28:44.399588 3289 catalog_manager.cc:4841] TS 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867): tablet b3bd90979e244f258a43f3d066868f98 (table test-table [id=12cda6359240433eb4f90f02a15e4a2c]) successfully deleted
I20260807 18:28:44.402931 3954 ts_tablet_manager.cc:1918] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:28:44.403110 3954 log.cc:1196] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/b3bd90979e244f258a43f3d066868f98
I20260807 18:28:44.403656 3954 ts_tablet_manager.cc:1939] T b3bd90979e244f258a43f3d066868f98 P 5b423204c1fc426e9470b84a0d35d458: Deleting consensus metadata
I20260807 18:28:44.404675 3290 catalog_manager.cc:4841] TS 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043): tablet b3bd90979e244f258a43f3d066868f98 (table test-table [id=12cda6359240433eb4f90f02a15e4a2c]) successfully deleted
W20260807 18:28:44.405172 3290 catalog_manager.cc:4808] TS 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043): delete failed for tablet b3bd90979e244f258a43f3d066868f98 because the tablet was not found. No further retry: Not found: Tablet not found: b3bd90979e244f258a43f3d066868f98
I20260807 18:28:44.405900 3859 tablet_bootstrap.cc:492] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458: No bootstrap required, opened a new log
I20260807 18:28:44.406072 3859 ts_tablet_manager.cc:1397] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.001s
I20260807 18:28:44.406749 3859 raft_consensus.cc:348] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.406953 3859 raft_consensus.cc:374] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:44.407066 3859 raft_consensus.cc:729] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b423204c1fc426e9470b84a0d35d458, State: Initialized, Role: FOLLOWER
I20260807 18:28:44.407353 3859 consensus_queue.cc:260] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.407836 3859 ts_tablet_manager.cc:1428] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 18:28:44.627336 3868 raft_consensus.cc:482] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:28:44.627558 3868 raft_consensus.cc:504] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.629057 3868 leader_election.cc:288] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043), 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635)
I20260807 18:28:44.635833 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.629481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:39955 (local address 127.27.1.195:45635)
0807 18:28:44.629710 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:44.629720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:44.630386 (+ 666us) server_negotiation.cc:407] Connection header received
0807 18:28:44.630758 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:44.630764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:44.630838 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:44.630939 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:44.631526 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.632236 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.634041 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.634814 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.635001 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.635078 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:44.635365 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 18:28:44.635537 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:28:44.635939 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.629378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:45635 (local address 127.27.1.193:39955)
0807 18:28:44.630264 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:44.630286 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:44.630516 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:44.630963 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:44.630969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:44.631077 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:44.631362 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:44.631371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.632382 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.632387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:44.633877 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:44.633893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.635002 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.635008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:44.635202 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.635226 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 18:28:44.635325 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 18:28:44.635528 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"spinlock_wait_cycles":7552,"thread_start_us":560,"threads_started":1}
I20260807 18:28:44.636224 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.629502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:56083 (local address 127.27.1.194:33043)
0807 18:28:44.630612 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:44.630618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:44.631072 (+ 454us) server_negotiation.cc:407] Connection header received
0807 18:28:44.631248 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:44.631255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:44.631325 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:44.631435 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:28:44.631953 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.632655 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.634344 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.635169 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.635360 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.635445 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:44.635738 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 18:28:44.635909 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":777,"threads_started":1}
I20260807 18:28:44.636590 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.629364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:33043 (local address 127.27.1.193:56083)
0807 18:28:44.630904 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:44.630926 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:44.631080 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:44.631452 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:44.631456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:44.631479 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:28:44.631797 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:44.631805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.632802 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.632807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:44.634201 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:44.634213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.635402 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.635408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:44.635579 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.635601 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 18:28:44.635713 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 18:28:44.635856 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"mutex_wait_us":26,"spinlock_wait_cycles":2944,"thread_start_us":1090,"threads_started":1}
I20260807 18:28:44.636647 3784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6479e3003394bdb8ff68562d61ec9dd" candidate_uuid: "56db8696eb2248fb927c7c02e8c60424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90ac215f4742d7adc29c062fadaa20" is_pre_election: true
I20260807 18:28:44.636925 3784 raft_consensus.cc:2454] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56db8696eb2248fb927c7c02e8c60424 in term 0.
I20260807 18:28:44.637295 3629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6479e3003394bdb8ff68562d61ec9dd" candidate_uuid: "56db8696eb2248fb927c7c02e8c60424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b423204c1fc426e9470b84a0d35d458" is_pre_election: true
I20260807 18:28:44.637431 3407 leader_election.cc:302] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56db8696eb2248fb927c7c02e8c60424, 8f90ac215f4742d7adc29c062fadaa20; no voters:
I20260807 18:28:44.637542 3629 raft_consensus.cc:2454] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56db8696eb2248fb927c7c02e8c60424 in term 0.
I20260807 18:28:44.637827 3868 raft_consensus.cc:2781] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:28:44.637962 3868 raft_consensus.cc:482] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:28:44.638087 3868 raft_consensus.cc:3037] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:44.640400 3868 raft_consensus.cc:504] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.641110 3868 leader_election.cc:288] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [CANDIDATE]: Term 1 election: Requested vote from peers 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043), 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635)
I20260807 18:28:44.641558 3629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6479e3003394bdb8ff68562d61ec9dd" candidate_uuid: "56db8696eb2248fb927c7c02e8c60424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b423204c1fc426e9470b84a0d35d458"
I20260807 18:28:44.641758 3629 raft_consensus.cc:3037] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:44.641772 3784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6479e3003394bdb8ff68562d61ec9dd" candidate_uuid: "56db8696eb2248fb927c7c02e8c60424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90ac215f4742d7adc29c062fadaa20"
I20260807 18:28:44.642019 3784 raft_consensus.cc:3037] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:44.643816 3629 raft_consensus.cc:2454] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56db8696eb2248fb927c7c02e8c60424 in term 1.
I20260807 18:28:44.644233 3406 leader_election.cc:302] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56db8696eb2248fb927c7c02e8c60424, 5b423204c1fc426e9470b84a0d35d458; no voters:
I20260807 18:28:44.644443 3784 raft_consensus.cc:2454] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56db8696eb2248fb927c7c02e8c60424 in term 1.
I20260807 18:28:44.644521 3868 raft_consensus.cc:2781] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:44.645012 3868 raft_consensus.cc:686] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 1 LEADER]: Becoming Leader. State: Replica: 56db8696eb2248fb927c7c02e8c60424, State: Running, Role: LEADER
I20260807 18:28:44.645395 3868 consensus_queue.cc:237] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } }
I20260807 18:28:44.648759 3305 catalog_manager.cc:5495] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193). New cstate: current_term: 1 leader_uuid: "56db8696eb2248fb927c7c02e8c60424" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56db8696eb2248fb927c7c02e8c60424" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 } health_report { overall_health: UNKNOWN } } }
I20260807 18:28:44.680500 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.674726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.27.1.193:34867)
0807 18:28:44.675327 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:44.675331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:44.675350 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:28:44.675426 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:44.675429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:44.675502 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:44.675638 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:44.676010 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.676775 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.678112 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.678582 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.678698 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.679174 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:44.679888 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:44.680024 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:44.680124 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 18:28:44.680207 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":355,"threads_started":1}
I20260807 18:28:44.695302 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.690259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:58507 (local address 127.27.1.194:33043)
0807 18:28:44.690542 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:44.690558 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:44.690927 (+ 369us) server_negotiation.cc:407] Connection header received
0807 18:28:44.691082 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:44.691086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:44.691206 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:44.691282 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:44.691724 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.692387 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.693665 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.694203 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.694297 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.694675 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:44.694785 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:44.694848 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:44.694984 (+ 136us) server_negotiation.cc:299] Negotiation successful
0807 18:28:44.695050 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:28:44.695560 3969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.690202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:33043 (local address 127.27.1.193:58507)
0807 18:28:44.690823 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:44.690844 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:44.690971 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:44.691308 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:44.691312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:44.691322 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 18:28:44.691599 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:44.691606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.692498 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.692502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:44.693541 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:44.693551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.694355 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.694359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:44.694484 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.694534 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 18:28:44.694868 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 18:28:44.694879 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 18:28:44.694947 (+ 68us) client_negotiation.cc:240] Negotiation successful
0807 18:28:44.695104 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":358,"threads_started":1}
I20260807 18:28:44.703480 3629 raft_consensus.cc:1261] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Refusing update from remote peer 56db8696eb2248fb927c7c02e8c60424: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:44.703554 3784 raft_consensus.cc:1261] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Refusing update from remote peer 56db8696eb2248fb927c7c02e8c60424: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:28:44.704222 3963 consensus_queue.cc:1035] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b423204c1fc426e9470b84a0d35d458" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:44.704572 3868 consensus_queue.cc:1035] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f90ac215f4742d7adc29c062fadaa20" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:28:44.731801 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.725996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:37247 (local address 127.27.1.193:34867)
0807 18:28:44.726198 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:44.726206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:44.726501 (+ 295us) server_negotiation.cc:407] Connection header received
0807 18:28:44.726716 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:44.726721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:44.726802 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:44.726906 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:28:44.727474 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.728275 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.729981 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.730470 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.730571 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.730951 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:28:44.731100 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:28:44.731207 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:44.731409 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 18:28:44.731487 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 18:28:44.732569 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.725815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34867 (local address 127.27.1.194:37247)
0807 18:28:44.726403 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:44.726418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:44.726575 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:44.726922 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:44.726927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:44.726938 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 18:28:44.727280 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:44.727290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.728401 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.728408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:44.729835 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:44.729849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.730624 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.730629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:44.730749 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.730784 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 18:28:44.731261 (+ 477us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 18:28:44.731273 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 18:28:44.732047 (+ 774us) client_negotiation.cc:240] Negotiation successful
0807 18:28:44.732187 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":284,"threads_started":1}
I20260807 18:28:44.767882 3305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:44.768165 3305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:28:44.771201 3305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fccbd6b85f704cc8b7cbaf820fb4e820: Sending DeleteTablet for 3 replicas of tablet b6479e3003394bdb8ff68562d61ec9dd
I20260807 18:28:44.771965 3458 tablet_service.cc:1514] Processing DeleteTablet for tablet b6479e3003394bdb8ff68562d61ec9dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:44 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:49898
I20260807 18:28:44.772122 3608 tablet_service.cc:1514] Processing DeleteTablet for tablet b6479e3003394bdb8ff68562d61ec9dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:44 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:38668
I20260807 18:28:44.772444 3952 tablet_replica.cc:330] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424: stopping tablet replica
I20260807 18:28:44.772440 3954 tablet_replica.cc:330] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458: stopping tablet replica
I20260807 18:28:44.772665 3952 raft_consensus.cc:2229] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:28:44.772672 3954 raft_consensus.cc:2229] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:44.772887 3954 raft_consensus.cc:2258] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:44.773166 3952 raft_consensus.cc:2258] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:44.773480 3756 tablet_service.cc:1514] Processing DeleteTablet for tablet b6479e3003394bdb8ff68562d61ec9dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:28:44 UTC) from {username='oryx', principal='oryx/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:60210
I20260807 18:28:44.773872 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 3386
I20260807 18:28:44.773931 3953 tablet_replica.cc:330] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20: stopping tablet replica
I20260807 18:28:44.774149 3953 raft_consensus.cc:2229] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:28:44.774348 3953 raft_consensus.cc:2258] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:28:44.775635 3954 ts_tablet_manager.cc:1905] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:44.775635 3952 ts_tablet_manager.cc:1905] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:44.782635 3954 ts_tablet_manager.cc:1918] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:28:44.782747 3952 ts_tablet_manager.cc:1918] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:28:44.782817 3954 log.cc:1196] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/b6479e3003394bdb8ff68562d61ec9dd
I20260807 18:28:44.782912 3952 log.cc:1196] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/b6479e3003394bdb8ff68562d61ec9dd
I20260807 18:28:44.783334 3954 ts_tablet_manager.cc:1939] T b6479e3003394bdb8ff68562d61ec9dd P 5b423204c1fc426e9470b84a0d35d458: Deleting consensus metadata
I20260807 18:28:44.783440 3952 ts_tablet_manager.cc:1939] T b6479e3003394bdb8ff68562d61ec9dd P 56db8696eb2248fb927c7c02e8c60424: Deleting consensus metadata
I20260807 18:28:44.784581 3289 catalog_manager.cc:4841] TS 56db8696eb2248fb927c7c02e8c60424 (127.27.1.193:34867): tablet b6479e3003394bdb8ff68562d61ec9dd (table test-table [id=3689de244dd84f4fbe14512241f6e5bf]) successfully deleted
I20260807 18:28:44.784590 3290 catalog_manager.cc:4841] TS 5b423204c1fc426e9470b84a0d35d458 (127.27.1.194:33043): tablet b6479e3003394bdb8ff68562d61ec9dd (table test-table [id=3689de244dd84f4fbe14512241f6e5bf]) successfully deleted
I20260807 18:28:44.785245 3953 ts_tablet_manager.cc:1905] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:28:44.789950 3953 ts_tablet_manager.cc:1918] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 18:28:44.790112 3953 log.cc:1196] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/b6479e3003394bdb8ff68562d61ec9dd
I20260807 18:28:44.790485 3953 ts_tablet_manager.cc:1939] T b6479e3003394bdb8ff68562d61ec9dd P 8f90ac215f4742d7adc29c062fadaa20: Deleting consensus metadata
I20260807 18:28:44.791661 3291 catalog_manager.cc:4841] TS 8f90ac215f4742d7adc29c062fadaa20 (127.27.1.195:45635): tablet b6479e3003394bdb8ff68562d61ec9dd (table test-table [id=3689de244dd84f4fbe14512241f6e5bf]) successfully deleted
Aug 07 18:28:44 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.193@KRBTEST.COM
I20260807 18:28:44.801844 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.775102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.27.1.193:34867)
0807 18:28:44.775327 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:44.775335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:44.776991 (+ 1656us) server_negotiation.cc:407] Connection header received
0807 18:28:44.777200 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:44.777219 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:44.777295 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:44.777427 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:44.778601 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.779475 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.781355 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.782094 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.782195 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.794852 (+ 12657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:44.794865 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:44.794884 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:44.794922 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:44.798474 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:44.799226 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:44.799236 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:44.799240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:44.799393 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:44.799788 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:44.799793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:44.799796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:44.800275 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:44.800587 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:44.801161 (+ 574us) server_negotiation.cc:299] Negotiation successful
0807 18:28:44.801425 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 18:28:44.802565 3528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:44.911901 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 3386
I20260807 18:28:44.936532 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 3537
Aug 07 18:28:44 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.194@KRBTEST.COM
I20260807 18:28:44.951246 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:44.937426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.27.1.194:33043)
0807 18:28:44.937598 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:44.937602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:44.937618 (+ 16us) server_negotiation.cc:407] Connection header received
0807 18:28:44.937675 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:44.937678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:44.937739 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:44.937828 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:44.939006 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.939784 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.940976 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:44.941380 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:44.941470 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:44.945551 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:44.945563 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:44.945578 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:44.945619 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:44.948706 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:44.949335 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:44.949343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:44.949345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:44.949493 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:44.949830 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:44.949835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:44.949839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:44.950232 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:44.950515 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:44.950692 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 18:28:44.950871 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:28:44.951933 3675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:45.102869 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 3537
I20260807 18:28:45.133378 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 3687
Aug 07 18:28:45 dist-test-slave-1xrh krb5kdc[3259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.1.195@KRBTEST.COM
I20260807 18:28:45.148578 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:45.134155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.27.1.195:45635)
0807 18:28:45.134364 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:45.134369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:45.134387 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:28:45.134551 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:45.134555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:45.134617 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:45.134698 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:45.136101 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:45.136852 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:45.138041 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:45.138424 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:45.138521 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:45.143192 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:45.143206 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:45.143222 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:45.143255 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:45.146127 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:45.146712 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:45.146718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:45.146721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:45.146853 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:45.147210 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:45.147213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:45.147215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:45.147546 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:45.147776 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:45.148069 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 18:28:45.148223 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:28:45.149281 3827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:45.275476 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 3687
I20260807 18:28:45.301726 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 3272
I20260807 18:28:45.302994 3350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:45.422387 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 3272
2026-08-07T18:28:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6233 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:28:45 dist-test-slave-1xrh krb5kdc[3983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:28:45 dist-test-slave-1xrh krb5kdc[3983](info): set up 2 sockets
Aug 07 18:28:45 dist-test-slave-1xrh krb5kdc[3983](info): commencing operation
krb5kdc: starting...
W20260807 18:28:46.582278 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.006s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:28:46 dist-test-slave-1xrh krb5kdc[3983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:28:46.835078 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:41031
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:47.166419 3997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:47.166695 3997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:47.166785 3997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:47.173897 3997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:47.174029 3997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:47.174108 3997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:47.174206 3997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:47.174300 3997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:47.184114 3997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:41031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:41031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:47.185372 3997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:47.186713 3997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:47.198729 4002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:47.198812 4003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:47.199026 4005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:47.199537 3997 server_base.cc:1034] running on GCE node
I20260807 18:28:47.200266 3997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:47.201655 3997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:47.202862 3997 hybrid_clock.cc:648] HybridClock initialized: now 1786127327202793 us; error 61 us; skew 500 ppm
I20260807 18:28:47.203235 3997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T18:28:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1849 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:28:47 dist-test-slave-1xrh krb5kdc[4011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:28:47 dist-test-slave-1xrh krb5kdc[4011](info): set up 2 sockets
Aug 07 18:28:47 dist-test-slave-1xrh krb5kdc[4011](info): commencing operation
krb5kdc: starting...
W20260807 18:28:48.418880 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:28:48 dist-test-slave-1xrh krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:28:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:28:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:28:48.674518 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44139
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:44139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:49.006645 4025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:49.006942 4025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:49.007107 4025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:49.014431 4025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:49.014536 4025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:49.014655 4025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:49.014752 4025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:49.014869 4025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:49.024801 4025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:44139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:49.026151 4025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:49.027446 4025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:49.039444 4034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:49.039508 4030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:49.039945 4032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:49.040223 4025 server_base.cc:1034] running on GCE node
I20260807 18:28:49.040894 4025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:49.042142 4025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:49.043371 4025 hybrid_clock.cc:648] HybridClock initialized: now 1786127329043326 us; error 53 us; skew 500 ppm
I20260807 18:28:49.043707 4025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:49 dist-test-slave-1xrh krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127329, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:49.046818 4025 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:28:49.048434 4025 webserver.cc:460] Webserver started at http://127.27.1.254:44363/ using document root <none> and password file <none>
I20260807 18:28:49.049119 4025 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:49.049263 4025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:49.049584 4025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:49.052574 4025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "02de4ecdfed8478d97a6be49613c8628"
format_stamp: "Formatted at 2026-08-07 18:28:49 on dist-test-slave-1xrh"
server_key: "62020a37e2c8412d355a45255f883f13"
server_key_iv: "25a591d4d39b542ba9c69bd0a3e3ec39"
server_key_version: "encryptionkey@0"
I20260807 18:28:49.053336 4025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "02de4ecdfed8478d97a6be49613c8628"
format_stamp: "Formatted at 2026-08-07 18:28:49 on dist-test-slave-1xrh"
server_key: "62020a37e2c8412d355a45255f883f13"
server_key_iv: "25a591d4d39b542ba9c69bd0a3e3ec39"
server_key_version: "encryptionkey@0"
I20260807 18:28:49.058434 4025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260807 18:28:49.061759 4042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:49.063022 4025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:28:49.063241 4025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "02de4ecdfed8478d97a6be49613c8628"
format_stamp: "Formatted at 2026-08-07 18:28:49 on dist-test-slave-1xrh"
server_key: "62020a37e2c8412d355a45255f883f13"
server_key_iv: "25a591d4d39b542ba9c69bd0a3e3ec39"
server_key_version: "encryptionkey@0"
I20260807 18:28:49.063450 4025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:49.084962 4025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:49.089419 4025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:49.089725 4025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:49.108263 4025 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:44139
I20260807 18:28:49.108299 4096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:44139 every 8 connection(s)
I20260807 18:28:49.109658 4025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:49.113194 4097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:28:49.119293 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 4025
I20260807 18:28:49.119535 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:28:49.120182 27655 external_mini_cluster.cc:1428] Setting key 4828201dc8e26b071f706f0f75a21539
I20260807 18:28:49.123448 4097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628: Bootstrap starting.
I20260807 18:28:49.126801 4097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628: Neither blocks nor log segments found. Creating new log.
I20260807 18:28:49.127964 4097 log.cc:824] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:49.131115 4097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628: No bootstrap required, opened a new log
Aug 07 18:28:49 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:49.140575 4097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } }
I20260807 18:28:49.140991 4097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:28:49.141181 4097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02de4ecdfed8478d97a6be49613c8628, State: Initialized, Role: FOLLOWER
I20260807 18:28:49.141767 4097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } }
I20260807 18:28:49.142125 4097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:49.142302 4097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:49.142516 4097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:28:49.144523 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:49.122142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.27.1.254:44139)
0807 18:28:49.122904 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:49.122915 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:49.122955 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:28:49.124779 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:49.124805 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:49.125156 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:49.125459 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:49.127359 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:49.128935 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:49.130716 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:49.131473 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:49.131600 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:49.136201 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:49.136211 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:49.136224 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:49.136262 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:49.139653 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:49.140326 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:49.140339 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:49.140346 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:49.140510 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:49.140919 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:49.140925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:49.140928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:49.141682 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:49.142009 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:49.142470 (+ 461us) server_negotiation.cc:299] Negotiation successful
0807 18:28:49.142952 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":361,"threads_started":1}
I20260807 18:28:49.145793 4097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } }
I20260807 18:28:49.146322 4097 leader_election.cc:302] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02de4ecdfed8478d97a6be49613c8628; no voters:
I20260807 18:28:49.146919 4097 leader_election.cc:288] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:28:49.147089 4103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:28:49.147959 4103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 1 LEADER]: Becoming Leader. State: Replica: 02de4ecdfed8478d97a6be49613c8628, State: Running, Role: LEADER
I20260807 18:28:49.148532 4103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } }
I20260807 18:28:49.149596 4097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:49.151739 4105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02de4ecdfed8478d97a6be49613c8628. Latest consensus state: current_term: 1 leader_uuid: "02de4ecdfed8478d97a6be49613c8628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } } }
I20260807 18:28:49.152222 4105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:49.153330 4104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02de4ecdfed8478d97a6be49613c8628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } } }
I20260807 18:28:49.153599 4104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:49.154129 4114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:49.158576 4114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:49.167343 4114 catalog_manager.cc:1347] Generated new cluster ID: 06db3ad5235a40b5a0fb10c1d8b4b1ab
I20260807 18:28:49.167456 4114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:49.185293 4114 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:28:49.187292 4114 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:49.199851 4114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628: Generated new TSK 0
I20260807 18:28:49.200889 4114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:28:49.275588 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44139
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:49.619009 4125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:49.619400 4125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:49.619606 4125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:49.626665 4125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:49.626763 4125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:49.626924 4125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:49.636480 4125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:49.637717 4125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:49.639037 4125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:49.652508 4131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:49.652578 4132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:49.652484 4134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:49.653760 4125 server_base.cc:1034] running on GCE node
I20260807 18:28:49.654377 4125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:49.655434 4125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:49.656679 4125 hybrid_clock.cc:648] HybridClock initialized: now 1786127329656629 us; error 41 us; skew 500 ppm
I20260807 18:28:49.657004 4125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:49 dist-test-slave-1xrh krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127329, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:49.660102 4125 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:28:49.661548 4125 webserver.cc:460] Webserver started at http://127.27.1.193:34375/ using document root <none> and password file <none>
I20260807 18:28:49.662209 4125 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:49.662352 4125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:49.662729 4125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:49.665683 4125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "a91a126a10be4e7e8d236a0a5b681267"
format_stamp: "Formatted at 2026-08-07 18:28:49 on dist-test-slave-1xrh"
server_key: "6c8345d7301512778614fbe7af64e363"
server_key_iv: "b0103ea695d8988c109d26f73bafb996"
server_key_version: "encryptionkey@0"
I20260807 18:28:49.666477 4125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "a91a126a10be4e7e8d236a0a5b681267"
format_stamp: "Formatted at 2026-08-07 18:28:49 on dist-test-slave-1xrh"
server_key: "6c8345d7301512778614fbe7af64e363"
server_key_iv: "b0103ea695d8988c109d26f73bafb996"
server_key_version: "encryptionkey@0"
I20260807 18:28:49.671578 4125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 18:28:49.674814 4142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:49.676108 4125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:49.676292 4125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "a91a126a10be4e7e8d236a0a5b681267"
format_stamp: "Formatted at 2026-08-07 18:28:49 on dist-test-slave-1xrh"
server_key: "6c8345d7301512778614fbe7af64e363"
server_key_iv: "b0103ea695d8988c109d26f73bafb996"
server_key_version: "encryptionkey@0"
I20260807 18:28:49.676488 4125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:49.693027 4125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:49.697103 4125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:49.697397 4125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:49.698354 4125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:49.700044 4125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:49.700148 4125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:49.700302 4125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:49.700389 4125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:49.734411 4125 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:34869
I20260807 18:28:49.734454 4264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:34869 every 8 connection(s)
I20260807 18:28:49.735793 4125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:49.741464 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 4125
I20260807 18:28:49.741690 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:28:49.742328 27655 external_mini_cluster.cc:1428] Setting key 46a96ffd1a3f385dac3ed1cd854ec949
Aug 07 18:28:49 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127329, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:49.758678 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:49.738314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:59137 (local address 127.27.1.254:44139)
0807 18:28:49.738760 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:49.738764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:49.739981 (+ 1217us) server_negotiation.cc:407] Connection header received
0807 18:28:49.741977 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:49.741981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:49.742051 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:49.742174 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:49.744216 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:49.745273 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:49.746891 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:49.747672 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:49.747813 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:49.751674 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:49.751683 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:49.751690 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:49.751737 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:49.755331 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:49.756330 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:49.756334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:49.756336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:49.756476 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:49.756980 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:49.756983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:49.756986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:49.757235 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:49.757344 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:49.758197 (+ 853us) server_negotiation.cc:299] Negotiation successful
0807 18:28:49.758413 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":230,"threads_started":1}
I20260807 18:28:49.760211 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:49.738912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:59137)
0807 18:28:49.739809 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:49.739855 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:49.741549 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:49.742487 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:49.742498 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:49.743201 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:49.743982 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:49.744008 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:49.745481 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:49.745485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:49.746737 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:49.746749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:49.747894 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:49.747899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:49.748112 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:49.748864 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:49.748886 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:49.751427 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:49.755468 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:49.755478 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:49.755499 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:49.756176 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:49.756598 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:49.756605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:49.756611 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:49.756870 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:49.757365 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:49.757376 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:49.757821 (+ 445us) client_negotiation.cc:769] Sending connection context
0807 18:28:49.758113 (+ 292us) client_negotiation.cc:240] Negotiation successful
0807 18:28:49.758703 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":401,"threads_started":1}
I20260807 18:28:49.761791 4265 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44139
I20260807 18:28:49.762089 4265 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:49.762804 4265 heartbeater.cc:511] Master 127.27.1.254:44139 requested a full tablet report, sending...
I20260807 18:28:49.764911 4061 ts_manager.cc:194] Registered new tserver with Master: a91a126a10be4e7e8d236a0a5b681267 (127.27.1.193:34869)
I20260807 18:28:49.766403 4061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:59137
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:28:49.804246 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44139
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:50.136442 4271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:50.136744 4271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:50.136919 4271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:50.144243 4271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:50.144402 4271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:50.144572 4271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:50.154420 4271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:50.155761 4271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:50.157152 4271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:50.170522 4277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:50.170519 4279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:50.170759 4276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:50.171545 4271 server_base.cc:1034] running on GCE node
I20260807 18:28:50.172108 4271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:50.173064 4271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:50.174330 4271 hybrid_clock.cc:648] HybridClock initialized: now 1786127330174270 us; error 66 us; skew 500 ppm
I20260807 18:28:50.174657 4271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:50 dist-test-slave-1xrh krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:50.183689 4271 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:28:50.185158 4271 webserver.cc:460] Webserver started at http://127.27.1.194:42953/ using document root <none> and password file <none>
I20260807 18:28:50.185848 4271 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:50.185952 4271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:50.186213 4271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:50.189154 4271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "41a230da9b1e43c7945b3101a0bf2cfe"
format_stamp: "Formatted at 2026-08-07 18:28:50 on dist-test-slave-1xrh"
server_key: "c1e278a916e3c0f59be6483ed76b7650"
server_key_iv: "5b7a0389482cf0143db817e4d68790cb"
server_key_version: "encryptionkey@0"
I20260807 18:28:50.189914 4271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "41a230da9b1e43c7945b3101a0bf2cfe"
format_stamp: "Formatted at 2026-08-07 18:28:50 on dist-test-slave-1xrh"
server_key: "c1e278a916e3c0f59be6483ed76b7650"
server_key_iv: "5b7a0389482cf0143db817e4d68790cb"
server_key_version: "encryptionkey@0"
I20260807 18:28:50.194979 4271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:28:50.198266 4288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:50.199631 4271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:50.199816 4271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "41a230da9b1e43c7945b3101a0bf2cfe"
format_stamp: "Formatted at 2026-08-07 18:28:50 on dist-test-slave-1xrh"
server_key: "c1e278a916e3c0f59be6483ed76b7650"
server_key_iv: "5b7a0389482cf0143db817e4d68790cb"
server_key_version: "encryptionkey@0"
I20260807 18:28:50.200006 4271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:50.221037 4271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:50.225186 4271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:50.225493 4271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:50.226490 4271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:50.228021 4271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:50.228098 4271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:50.228267 4271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:50.228317 4271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:50.261966 4271 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:43951
I20260807 18:28:50.262023 4413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:43951 every 8 connection(s)
I20260807 18:28:50.263342 4271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:50.268296 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 4271
I20260807 18:28:50.268499 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:28:50.269050 27655 external_mini_cluster.cc:1428] Setting key ebc852833cc9eadfb1cc6214fd415c7a
Aug 07 18:28:50 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:50.285781 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:50.265878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:56669 (local address 127.27.1.254:44139)
0807 18:28:50.266356 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:50.266361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:50.267347 (+ 986us) server_negotiation.cc:407] Connection header received
0807 18:28:50.269886 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:50.269892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:50.269980 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:50.270076 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:50.271909 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:50.272633 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:50.275117 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:50.275607 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:50.275702 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:50.279830 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:50.279835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:50.279838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:50.279870 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:50.282700 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:50.283502 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:50.283512 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:50.283514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:50.283647 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:50.284109 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:50.284114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:50.284115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:50.284359 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:50.284521 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:50.285167 (+ 646us) server_negotiation.cc:299] Negotiation successful
0807 18:28:50.285396 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":264,"threads_started":1}
I20260807 18:28:50.286703 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:50.266367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:56669)
0807 18:28:50.267177 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:50.267228 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:50.269577 (+ 2349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:50.270322 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:50.270329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:50.270935 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:50.271729 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:50.271757 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:50.272800 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:50.272808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:50.274956 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:50.274985 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:50.275836 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:50.275845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:50.276160 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:50.276861 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:50.276899 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:50.279604 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:50.282842 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:50.282851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:50.282866 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:50.283374 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:50.283763 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:50.283768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:50.283772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:50.283980 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:50.284494 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:50.284501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:50.284813 (+ 312us) client_negotiation.cc:769] Sending connection context
0807 18:28:50.285077 (+ 264us) client_negotiation.cc:240] Negotiation successful
0807 18:28:50.285575 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":262,"threads_started":1}
I20260807 18:28:50.288259 4414 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44139
I20260807 18:28:50.288569 4414 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:50.289258 4414 heartbeater.cc:511] Master 127.27.1.254:44139 requested a full tablet report, sending...
I20260807 18:28:50.290787 4061 ts_manager.cc:194] Registered new tserver with Master: 41a230da9b1e43c7945b3101a0bf2cfe (127.27.1.194:43951)
I20260807 18:28:50.291652 4061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:56669
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:28:50.328352 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44139
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:50.657739 4420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:50.658039 4420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:50.658206 4420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:50.665267 4420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:50.665410 4420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:50.665576 4420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:50.674947 4420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:50.676206 4420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:50.677484 4420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:50.690723 4428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:50.690681 4425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:50.690769 4426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:50.696519 4420 server_base.cc:1034] running on GCE node
I20260807 18:28:50.697120 4420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:50.698060 4420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:50.699281 4420 hybrid_clock.cc:648] HybridClock initialized: now 1786127330699238 us; error 50 us; skew 500 ppm
I20260807 18:28:50.699611 4420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:50 dist-test-slave-1xrh krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:50.702770 4420 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:28:50.704596 4420 webserver.cc:460] Webserver started at http://127.27.1.195:42167/ using document root <none> and password file <none>
I20260807 18:28:50.705346 4420 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:50.705503 4420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:50.705813 4420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:28:50.708756 4420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "455b6a26b9d94b38a3d7b68a741b991d"
format_stamp: "Formatted at 2026-08-07 18:28:50 on dist-test-slave-1xrh"
server_key: "127ebc0596bb7242df015cfe6503723f"
server_key_iv: "b5b4f3e9bb01052b79822f048d92a0e2"
server_key_version: "encryptionkey@0"
I20260807 18:28:50.709548 4420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "455b6a26b9d94b38a3d7b68a741b991d"
format_stamp: "Formatted at 2026-08-07 18:28:50 on dist-test-slave-1xrh"
server_key: "127ebc0596bb7242df015cfe6503723f"
server_key_iv: "b5b4f3e9bb01052b79822f048d92a0e2"
server_key_version: "encryptionkey@0"
I20260807 18:28:50.714440 4420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 18:28:50.717787 4436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:50.719048 4420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:28:50.719247 4420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "455b6a26b9d94b38a3d7b68a741b991d"
format_stamp: "Formatted at 2026-08-07 18:28:50 on dist-test-slave-1xrh"
server_key: "127ebc0596bb7242df015cfe6503723f"
server_key_iv: "b5b4f3e9bb01052b79822f048d92a0e2"
server_key_version: "encryptionkey@0"
I20260807 18:28:50.719455 4420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:50.735960 4420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:50.740115 4420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:50.740401 4420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:50.741317 4420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:50.742592 4420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:50.742713 4420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:50.742862 4420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:50.742934 4420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:50.769304 4265 heartbeater.cc:503] Master 127.27.1.254:44139 was elected leader, sending a full tablet report...
I20260807 18:28:50.776860 4420 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:41209
I20260807 18:28:50.776895 4558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:41209 every 8 connection(s)
I20260807 18:28:50.778158 4420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:50.782953 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 4420
I20260807 18:28:50.783171 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:28:50.783730 27655 external_mini_cluster.cc:1428] Setting key 3854962fbc915868f52b76d44f295815
Aug 07 18:28:50 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:28:50.799454 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:50.780781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:46913 (local address 127.27.1.254:44139)
0807 18:28:50.780957 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:50.780961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:50.782165 (+ 1204us) server_negotiation.cc:407] Connection header received
0807 18:28:50.784706 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:50.784711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:50.784829 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:50.784939 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:50.786615 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:50.787641 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:50.789311 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:50.789693 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:50.789822 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:50.793245 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:50.793250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:50.793253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:50.793285 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:50.795941 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:50.796663 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:50.796668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:50.796673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:50.796787 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:50.797400 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:50.797404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:50.797407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:50.797783 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:50.798005 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:50.798881 (+ 876us) server_negotiation.cc:299] Negotiation successful
0807 18:28:50.799157 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 18:28:50.800403 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:50.781251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:46913)
0807 18:28:50.782010 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:50.782047 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:50.784434 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:50.785201 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:50.785213 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:50.785845 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:50.786444 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:50.786461 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:50.787805 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:50.787809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:50.789075 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:50.789109 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:50.789937 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:50.789945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:50.790165 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:50.790850 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:50.790877 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:50.793036 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:28:50.796088 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:50.796096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:50.796113 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:50.796537 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:50.796896 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:28:50.796901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:28:50.796905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:28:50.797157 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:28:50.797955 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:28:50.797965 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:28:50.798406 (+ 441us) client_negotiation.cc:769] Sending connection context
0807 18:28:50.798774 (+ 368us) client_negotiation.cc:240] Negotiation successful
0807 18:28:50.799302 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":285,"threads_started":1}
I20260807 18:28:50.801674 4559 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44139
I20260807 18:28:50.801985 4559 heartbeater.cc:464] Registering TS with master...
I20260807 18:28:50.802747 4559 heartbeater.cc:511] Master 127.27.1.254:44139 requested a full tablet report, sending...
I20260807 18:28:50.804142 4061 ts_manager.cc:194] Registered new tserver with Master: 455b6a26b9d94b38a3d7b68a741b991d (127.27.1.195:41209)
I20260807 18:28:50.804920 4061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:46913
I20260807 18:28:50.811348 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab.
Entry for principal oryx/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab.
WARNING: no policy specified for oryx/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab.
Entry for principal oryx/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab.
WARNING: no policy specified for oryx/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab.
Entry for principal oryx/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab.
I20260807 18:28:50.969455 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4125
Aug 07 18:28:50 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:28:50.985829 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:50.970578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.27.1.193:34869)
0807 18:28:50.971550 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:50.971557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:50.971591 (+ 34us) server_negotiation.cc:407] Connection header received
0807 18:28:50.971653 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:50.971660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:50.971869 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:50.972001 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:50.973330 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:50.974340 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:50.975653 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:50.976200 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:50.976299 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:50.980379 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:50.980390 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:50.980404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:50.980443 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:50.983455 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:50.984072 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:50.984079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:50.984081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:50.984201 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:50.984507 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:50.984510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:50.984512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:50.984828 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:50.985045 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:50.985349 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 18:28:50.985549 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":629,"threads_started":1}
I20260807 18:28:50.987177 4259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:51.069590 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 4125
I20260807 18:28:51.088697 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4271
Aug 07 18:28:51 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:28:51.104768 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:51.089597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.27.1.194:43951)
0807 18:28:51.090173 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:51.090180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:51.090207 (+ 27us) server_negotiation.cc:407] Connection header received
0807 18:28:51.090271 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:51.090278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:51.090510 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:51.090679 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:51.091994 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:51.093019 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:51.094300 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:51.094824 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:51.094917 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:51.099066 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:51.099076 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:51.099091 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:51.099172 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:51.102075 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:51.102769 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:51.102776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:51.102778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:51.102909 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:51.103335 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:51.103339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:51.103341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:51.103670 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:51.103910 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:51.104228 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 18:28:51.104476 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":297,"threads_started":1}
I20260807 18:28:51.105993 4409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:51.216012 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 4271
I20260807 18:28:51.235812 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4420
Aug 07 18:28:51 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:28:51.254464 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:51.236831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.27.1.195:41209)
0807 18:28:51.239561 (+ 2730us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:51.239569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:51.239600 (+ 31us) server_negotiation.cc:407] Connection header received
0807 18:28:51.239667 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:51.239672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:51.239913 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:51.240080 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:51.241351 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:51.242344 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:51.243752 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:51.244323 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:51.244431 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:51.248844 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:51.248855 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:51.248869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:51.248904 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:51.251965 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:51.252593 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:51.252599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:51.252601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:51.252718 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:51.253051 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:51.253054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:51.253056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:51.253399 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:51.253614 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:51.253830 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 18:28:51.254020 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2583,"thread_start_us":2438,"threads_started":1}
I20260807 18:28:51.256018 4553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:51.345446 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 4420
I20260807 18:28:51.364886 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4025
I20260807 18:28:51.366107 4092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:28:51.472056 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 4025
I20260807 18:28:51.496155 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44139
--webserver_interface=127.27.1.254
--webserver_port=44363
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:44139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:28:51.823935 4579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:51.824280 4579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:51.824447 4579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:51.831421 4579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:28:51.831563 4579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:51.831672 4579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:51.831779 4579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:28:51.831866 4579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:28:51.841250 4579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:44139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=44363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:51.842533 4579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:51.843827 4579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:51.855176 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:51.855108 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:51.855233 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:51.856112 4579 server_base.cc:1034] running on GCE node
I20260807 18:28:51.856827 4579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:51.858177 4579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:51.859397 4579 hybrid_clock.cc:648] HybridClock initialized: now 1786127331859358 us; error 53 us; skew 500 ppm
I20260807 18:28:51.859721 4579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:51 dist-test-slave-1xrh krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:51.862633 4579 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:28:51.864063 4579 webserver.cc:460] Webserver started at http://127.27.1.254:44363/ using document root <none> and password file <none>
I20260807 18:28:51.864670 4579 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:51.864784 4579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:51.869884 4579 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260807 18:28:51.872543 4596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:51.873785 4579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 18:28:51.873919 4579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "02de4ecdfed8478d97a6be49613c8628"
format_stamp: "Formatted at 2026-08-07 18:28:49 on dist-test-slave-1xrh"
server_key: "62020a37e2c8412d355a45255f883f13"
server_key_iv: "25a591d4d39b542ba9c69bd0a3e3ec39"
server_key_version: "encryptionkey@0"
I20260807 18:28:51.874804 4579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:51.895325 4579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:51.899236 4579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:51.899458 4579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:51.918116 4579 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:44139
I20260807 18:28:51.918139 4648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:44139 every 8 connection(s)
I20260807 18:28:51.919662 4579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:28:51.921372 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 4579
I20260807 18:28:51.922403 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:34869
--local_ip_for_outbound_sockets=127.27.1.193
--tserver_master_addrs=127.27.1.254:44139
--webserver_port=34375
--webserver_interface=127.27.1.193
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.1.193
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:28:51.929327 4649 sys_catalog.cc:263] Verifying existing consensus state
I20260807 18:28:51.934166 4649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628: Bootstrap starting.
I20260807 18:28:51.954066 4649 log.cc:824] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628: Log is configured to *not* fsync() on all Append() calls
I20260807 18:28:51.963405 4649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 18:28:51.963989 4649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628: Bootstrap complete.
I20260807 18:28:51.973264 4649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } }
I20260807 18:28:51.974112 4649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02de4ecdfed8478d97a6be49613c8628, State: Initialized, Role: FOLLOWER
I20260807 18:28:51.974777 4649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } }
I20260807 18:28:51.975020 4649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:28:51.975167 4649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:28:51.975378 4649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 1 FOLLOWER]: Advancing to term 2
I20260807 18:28:51.979763 4649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } }
I20260807 18:28:51.980291 4649 leader_election.cc:302] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02de4ecdfed8478d97a6be49613c8628; no voters:
I20260807 18:28:51.980990 4649 leader_election.cc:288] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 18:28:51.981142 4653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 18:28:51.982434 4653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [term 2 LEADER]: Becoming Leader. State: Replica: 02de4ecdfed8478d97a6be49613c8628, State: Running, Role: LEADER
I20260807 18:28:51.983084 4653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } }
I20260807 18:28:51.983397 4649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:28:51.985754 4655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02de4ecdfed8478d97a6be49613c8628. Latest consensus state: current_term: 2 leader_uuid: "02de4ecdfed8478d97a6be49613c8628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } } }
I20260807 18:28:51.986135 4655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:51.986733 4654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "02de4ecdfed8478d97a6be49613c8628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02de4ecdfed8478d97a6be49613c8628" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44139 } } }
I20260807 18:28:51.987039 4654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628 [sys.catalog]: This master's current role is: LEADER
I20260807 18:28:51.992141 4661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:28:51.998773 4661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:28:52.000247 4661 catalog_manager.cc:1259] Loaded cluster ID: 06db3ad5235a40b5a0fb10c1d8b4b1ab
I20260807 18:28:52.000403 4661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:28:52.004467 4661 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:28:52.005986 4661 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 02de4ecdfed8478d97a6be49613c8628: Loaded TSK: 0
I20260807 18:28:52.007627 4661 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 18:28:52.272660 4651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:52.272944 4651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:52.273103 4651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:52.280624 4651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:52.280730 4651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:52.280895 4651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:28:52.290822 4651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.193.keytab
--principal=oryx/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:34869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=34375
--tserver_master_addrs=127.27.1.254:44139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:52.292304 4651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:52.293774 4651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:52.307425 4679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:52.307560 4678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:52.307593 4681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:52.308183 4651 server_base.cc:1034] running on GCE node
I20260807 18:28:52.308784 4651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:52.309926 4651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:52.311197 4651 hybrid_clock.cc:648] HybridClock initialized: now 1786127332311078 us; error 61 us; skew 500 ppm
I20260807 18:28:52.311523 4651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127332, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:52.314738 4651 init.cc:377] Logged in from keytab as oryx/127.27.1.193@KRBTEST.COM (short username oryx)
I20260807 18:28:52.316371 4651 webserver.cc:460] Webserver started at http://127.27.1.193:34375/ using document root <none> and password file <none>
I20260807 18:28:52.317085 4651 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:52.317233 4651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:52.322916 4651 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 18:28:52.325802 4691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:52.327482 4651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:52.327656 4651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "a91a126a10be4e7e8d236a0a5b681267"
format_stamp: "Formatted at 2026-08-07 18:28:49 on dist-test-slave-1xrh"
server_key: "6c8345d7301512778614fbe7af64e363"
server_key_iv: "b0103ea695d8988c109d26f73bafb996"
server_key_version: "encryptionkey@0"
I20260807 18:28:52.328697 4651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:52.348058 4651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:52.352254 4651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:52.352540 4651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:52.353500 4651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:52.354760 4651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:52.354856 4651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:52.355005 4651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:52.355122 4651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:52.387723 4651 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:34869
I20260807 18:28:52.387789 4814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:34869 every 8 connection(s)
I20260807 18:28:52.389222 4651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:28:52.391019 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 4651
I20260807 18:28:52.391705 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:43951
--local_ip_for_outbound_sockets=127.27.1.194
--tserver_master_addrs=127.27.1.254:44139
--webserver_port=42953
--webserver_interface=127.27.1.194
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.1.194
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:52.410440 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:52.392542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:41795)
0807 18:28:52.393353 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:52.393420 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:52.396383 (+ 2963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:52.397861 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:52.397876 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:52.398688 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:52.399544 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.399573 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.401203 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.401211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.403103 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.403116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.404687 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.404692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.404905 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.405645 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:52.405675 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:52.409459 (+ 3784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":978,"threads_started":1}
W20260807 18:28:52.411095 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:52.393254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:41795 (local address 127.27.1.254:44139)
0807 18:28:52.394024 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:52.394038 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:52.394089 (+ 51us) server_negotiation.cc:407] Connection header received
0807 18:28:52.396991 (+ 2902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:52.397005 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:52.397322 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:52.397634 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:52.399781 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.401022 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.403304 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.404451 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.404640 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.409903 (+ 5263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:41795: BlockingRecv error: recv got EOF from 127.27.1.193:41795 (error 108)
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":967,"threads_started":1}
W20260807 18:28:52.411155 4815 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:44139 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.1.254:44139: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:52.419750 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:52.411749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:48553)
0807 18:28:52.411920 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:52.411946 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:52.412052 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:52.412447 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:52.412451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:52.412791 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:52.412923 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.412930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.414011 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.414016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.414898 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.414919 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.415665 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.415671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.415856 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.416512 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:52.416530 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:52.419453 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 18:28:52.420239 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:52.411987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:48553 (local address 127.27.1.254:44139)
0807 18:28:52.412187 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:52.412200 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:52.412214 (+ 14us) server_negotiation.cc:407] Connection header received
0807 18:28:52.412252 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:52.412256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:52.412316 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:52.412423 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:52.413064 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.413869 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.415062 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.415518 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.415653 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.419661 (+ 4008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:48553: BlockingRecv error: recv got EOF from 127.27.1.193:48553 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:52.427953 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:52.420422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:44239)
0807 18:28:52.420603 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:52.420617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:52.420690 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:52.421103 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:52.421107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:52.421398 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:52.421531 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.421538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.422492 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.422498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.423452 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.423463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.424151 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.424157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.424343 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.424919 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:52.424936 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:52.427659 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260807 18:28:52.428229 4815 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 18:28:52.428337 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:52.420585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:44239 (local address 127.27.1.254:44139)
0807 18:28:52.420850 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:52.420854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:52.420866 (+ 12us) server_negotiation.cc:407] Connection header received
0807 18:28:52.420902 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:52.420905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:52.420971 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:52.421076 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:52.421673 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.422345 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.423598 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.424005 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.424141 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.427856 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:44239: BlockingRecv error: recv got EOF from 127.27.1.193:44239 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
W20260807 18:28:52.729281 4819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:52.729630 4819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:52.729795 4819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:52.737154 4819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:52.737303 4819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:52.737488 4819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:28:52.746958 4819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.194.keytab
--principal=oryx/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:43951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=42953
--tserver_master_addrs=127.27.1.254:44139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:52.748322 4819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:52.749712 4819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:52.763207 4828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:52.763262 4826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:52.763463 4824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:52.763967 4819 server_base.cc:1034] running on GCE node
I20260807 18:28:52.764540 4819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:52.765491 4819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:52.766723 4819 hybrid_clock.cc:648] HybridClock initialized: now 1786127332766675 us; error 43 us; skew 500 ppm
I20260807 18:28:52.767052 4819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127332, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:52.770262 4819 init.cc:377] Logged in from keytab as oryx/127.27.1.194@KRBTEST.COM (short username oryx)
I20260807 18:28:52.771741 4819 webserver.cc:460] Webserver started at http://127.27.1.194:42953/ using document root <none> and password file <none>
I20260807 18:28:52.772372 4819 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:52.772485 4819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:52.777731 4819 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260807 18:28:52.780529 4836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:52.781831 4819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:28:52.782017 4819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "41a230da9b1e43c7945b3101a0bf2cfe"
format_stamp: "Formatted at 2026-08-07 18:28:50 on dist-test-slave-1xrh"
server_key: "c1e278a916e3c0f59be6483ed76b7650"
server_key_iv: "5b7a0389482cf0143db817e4d68790cb"
server_key_version: "encryptionkey@0"
I20260807 18:28:52.782963 4819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:52.802830 4819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:52.807271 4819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:52.807540 4819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:52.808516 4819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:52.809846 4819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:52.809970 4819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:52.810109 4819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:52.810214 4819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:52.844324 4819 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:43951
I20260807 18:28:52.844417 4959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:43951 every 8 connection(s)
I20260807 18:28:52.845775 4819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:28:52.851157 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 4819
I20260807 18:28:52.851830 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:41209
--local_ip_for_outbound_sockets=127.27.1.195
--tserver_master_addrs=127.27.1.254:44139
--webserver_port=42167
--webserver_interface=127.27.1.195
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.1.195
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
W20260807 18:28:52.862043 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:52.848413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:55869 (local address 127.27.1.254:44139)
0807 18:28:52.848666 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:52.848671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:52.849730 (+ 1059us) server_negotiation.cc:407] Connection header received
0807 18:28:52.851728 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:52.851734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:52.851825 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:52.851941 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:52.853423 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.854139 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.855749 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.856172 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.856308 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.861754 (+ 5446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:55869: BlockingRecv error: recv got EOF from 127.27.1.194:55869 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 18:28:52.862475 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:52.848862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:55869)
0807 18:28:52.849559 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:52.849601 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:52.851451 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:52.852187 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:52.852195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:52.852707 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:52.853243 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.853259 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.854316 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.854320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.855604 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.855617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.856426 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.856433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.856655 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.857340 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:52.857367 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:52.861473 (+ 4106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":275,"threads_started":1}
W20260807 18:28:52.863209 4960 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:44139 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.1.254:44139: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:52.871517 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:52.863820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:49257)
0807 18:28:52.863981 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:52.864009 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:52.864106 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:52.864405 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:52.864410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:52.864729 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:52.864854 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.864862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.865899 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.865912 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.866770 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.866781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.867600 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.867606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.867769 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.868352 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:52.868368 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:52.871226 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260807 18:28:52.871666 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:52.863931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:49257 (local address 127.27.1.254:44139)
0807 18:28:52.864099 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:52.864105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:52.864123 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:28:52.864205 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:52.864210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:52.864280 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:52.864381 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:52.865005 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.865741 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.866958 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.867440 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.867577 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.871417 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:49257: BlockingRecv error: recv got EOF from 127.27.1.194:49257 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:52 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:52.881675 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:52.872332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:38281)
0807 18:28:52.872550 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:52.872572 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:52.872680 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:52.872979 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:52.872987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:52.873424 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:52.873659 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.873675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.874603 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.874611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.876357 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:52.876377 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.877057 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.877066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:52.877344 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.878021 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:52.878049 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:52.881372 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 18:28:52.881891 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:52.872509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:38281 (local address 127.27.1.254:44139)
0807 18:28:52.872673 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:52.872677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:52.872692 (+ 15us) server_negotiation.cc:407] Connection header received
0807 18:28:52.872788 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:52.872791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:52.872840 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:52.872920 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:52.873800 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.874455 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.876511 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:52.876895 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:52.876996 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:52.881558 (+ 4562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:38281: BlockingRecv error: recv got EOF from 127.27.1.194:38281 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260807 18:28:52.881950 4960 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 18:28:53.194911 4964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:28:53.195258 4964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:28:53.195441 4964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:28:53.202522 4964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:28:53.202672 4964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:28:53.202864 4964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:28:53.212558 4964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/krb5kdc/oryx_127.27.1.195.keytab
--principal=oryx/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:41209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=42167
--tserver_master_addrs=127.27.1.254:44139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:28:53.213862 4964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:28:53.215255 4964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:28:53.228541 4969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:53.228541 4972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:28:53.228535 4970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:28:53.229463 4964 server_base.cc:1034] running on GCE node
I20260807 18:28:53.230012 4964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:28:53.230916 4964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:28:53.232156 4964 hybrid_clock.cc:648] HybridClock initialized: now 1786127333232087 us; error 75 us; skew 500 ppm
I20260807 18:28:53.232532 4964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127333, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:28:53.235673 4964 init.cc:377] Logged in from keytab as oryx/127.27.1.195@KRBTEST.COM (short username oryx)
I20260807 18:28:53.236997 4964 webserver.cc:460] Webserver started at http://127.27.1.195:42167/ using document root <none> and password file <none>
I20260807 18:28:53.237618 4964 fs_manager.cc:359] Metadata directory not provided
I20260807 18:28:53.237730 4964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:28:53.242985 4964 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260807 18:28:53.245609 4981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:53.246932 4964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:28:53.247112 4964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "455b6a26b9d94b38a3d7b68a741b991d"
format_stamp: "Formatted at 2026-08-07 18:28:50 on dist-test-slave-1xrh"
server_key: "127ebc0596bb7242df015cfe6503723f"
server_key_iv: "b5b4f3e9bb01052b79822f048d92a0e2"
server_key_version: "encryptionkey@0"
I20260807 18:28:53.248134 4964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:28:53.266793 4964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:28:53.271311 4964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:28:53.271588 4964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:28:53.272585 4964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:28:53.273921 4964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:28:53.274044 4964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:53.274175 4964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:28:53.274260 4964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:28:53.308799 4964 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:41209
I20260807 18:28:53.308893 5110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:41209 every 8 connection(s)
I20260807 18:28:53.310809 4964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:28:53.319646 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 4964
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
W20260807 18:28:53.332001 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:53.316158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:56673 (local address 127.27.1.254:44139)
0807 18:28:53.316660 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:53.316669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:53.317216 (+ 547us) server_negotiation.cc:407] Connection header received
0807 18:28:53.319574 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:53.319579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:53.319669 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:53.319804 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:53.322307 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.323047 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.324697 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.325342 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.325475 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.331791 (+ 6316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:56673: BlockingRecv error: recv got EOF from 127.27.1.195:56673 (error 108)
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":323}
I20260807 18:28:53.332990 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:53.315595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:56673)
0807 18:28:53.317033 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:53.317095 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:53.319289 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:53.320179 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:53.320192 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:53.321092 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:53.322071 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:53.322101 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.323237 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.323245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:53.324541 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:53.324553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.325586 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.325596 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:53.325920 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.326674 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:53.326708 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:53.331632 (+ 4924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":357,"threads_started":1}
W20260807 18:28:53.333789 5111 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:44139 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.1.254:44139: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
I20260807 18:28:53.339732 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:53.320717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.27.1.254:44139)
0807 18:28:53.323472 (+ 2755us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:53.323476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:53.323489 (+ 13us) server_negotiation.cc:407] Connection header received
0807 18:28:53.323525 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:53.323528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:53.323576 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:53.323693 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:53.325468 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.326663 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.328664 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.329231 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.329369 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.332653 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:28:53.332661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:28:53.332676 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:28:53.332734 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:28:53.336615 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:53.337291 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:53.337300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:53.337308 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:53.337507 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:28:53.337880 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:28:53.337885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:28:53.337889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:28:53.338317 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:28:53.338677 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0807 18:28:53.339074 (+ 397us) server_negotiation.cc:299] Negotiation successful
0807 18:28:53.339460 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":2661,"thread_start_us":2560,"threads_started":1}
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:53.342993 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:53.334410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:38605)
0807 18:28:53.334822 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:53.334838 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:53.334927 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:53.335241 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:53.335244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:53.335558 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:53.335688 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:53.335695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.337057 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.337062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:53.337925 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:53.337936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.338687 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.338693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:53.338856 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.339541 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:53.339556 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:53.342815 (+ 3259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20260807 18:28:53.343622 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:53.334536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:38605 (local address 127.27.1.254:44139)
0807 18:28:53.334688 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:53.334712 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:53.334919 (+ 207us) server_negotiation.cc:407] Connection header received
0807 18:28:53.335010 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:53.335016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:53.335102 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:53.335228 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:53.335832 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.336928 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.338066 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.338536 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.338672 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.343030 (+ 4358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:38605: BlockingRecv error: recv got EOF from 127.27.1.195:38605 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:53.350950 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:53.343738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:38385)
0807 18:28:53.343982 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:53.343996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:53.344068 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:53.344327 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:53.344329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:53.344603 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:53.344743 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:53.344751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.345687 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.345691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:53.346592 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:53.346603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.347291 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.347300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:53.347546 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.348065 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:53.348081 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:53.350646 (+ 2565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135}
W20260807 18:28:53.351076 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:53.343841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:38385 (local address 127.27.1.254:44139)
0807 18:28:53.344046 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:53.344050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:53.344065 (+ 15us) server_negotiation.cc:407] Connection header received
0807 18:28:53.344153 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:53.344156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:53.344216 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:53.344290 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:53.344862 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.345556 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.346712 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.347096 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.347257 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.350828 (+ 3571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:38385: BlockingRecv error: recv got EOF from 127.27.1.195:38385 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260807 18:28:53.351291 5111 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:53.437969 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:53.428817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:33139)
0807 18:28:53.429444 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:53.429460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:53.429632 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:53.430053 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:53.430058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:53.430580 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:53.430776 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:53.430786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.431959 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.431963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:53.432953 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:53.432966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.433739 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.433746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:53.433923 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.434495 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:53.434512 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:53.437550 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":357,"threads_started":1}
W20260807 18:28:53.438022 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:53.428998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:33139 (local address 127.27.1.254:44139)
0807 18:28:53.429248 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:53.429263 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:53.429558 (+ 295us) server_negotiation.cc:407] Connection header received
0807 18:28:53.429765 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:53.429772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:53.429897 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:53.430018 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:53.430961 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.431746 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.433102 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.433545 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.433676 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.437781 (+ 4105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:33139: BlockingRecv error: recv got EOF from 127.27.1.193:33139 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:53 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:53.891737 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:53.882524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:51783)
0807 18:28:53.883096 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:53.883113 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:53.883292 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:53.883684 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:53.883691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:53.884211 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:53.884430 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:53.884440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.885553 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.885559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:53.886527 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:53.886539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.887364 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.887372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:53.887542 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.888221 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:53.888239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:53.891380 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":286,"threads_started":1}
W20260807 18:28:53.891812 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:53.882605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:51783 (local address 127.27.1.254:44139)
0807 18:28:53.882811 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:53.882820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:53.883222 (+ 402us) server_negotiation.cc:407] Connection header received
0807 18:28:53.883443 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:53.883448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:53.883526 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:53.883634 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:53.884583 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.885408 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.886686 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:53.887115 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:53.887295 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:53.891564 (+ 4269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:51783: BlockingRecv error: recv got EOF from 127.27.1.194:51783 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 18:28:54 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:54 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:54.361043 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:54.352100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:37933)
0807 18:28:54.352671 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:54.352688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:54.352807 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:54.353110 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:54.353113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:54.353592 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:54.353865 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:54.353876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.354867 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.354871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:54.356065 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:54.356079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.356745 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.356749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:54.356892 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:54.357448 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:54.357465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:54.360689 (+ 3224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":326,"threads_started":1}
W20260807 18:28:54.361129 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:54.352092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:37933 (local address 127.27.1.254:44139)
0807 18:28:54.352259 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:54.352263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:54.352766 (+ 503us) server_negotiation.cc:407] Connection header received
0807 18:28:54.352916 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:54.352919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:54.352980 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:54.353061 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:54.353997 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.354714 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.356198 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.356589 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.356680 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:54.360899 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:37933: BlockingRecv error: recv got EOF from 127.27.1.195:37933 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 07 18:28:54 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:54 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:54.447691 5120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:54.438836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:49643)
0807 18:28:54.439386 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:54.439402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:54.439556 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:54.439912 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:54.439917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:54.440342 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:54.440487 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:54.440498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.441523 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.441530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:54.442440 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:54.442452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.443177 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.443184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:54.443344 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:54.444013 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:54.444032 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:54.447440 (+ 3408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":329,"threads_started":1}
W20260807 18:28:54.447894 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:54.438891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:49643 (local address 127.27.1.254:44139)
0807 18:28:54.439111 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:54.439119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:54.439491 (+ 372us) server_negotiation.cc:407] Connection header received
0807 18:28:54.439679 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:54.439691 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:54.439753 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:54.439857 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:54.440628 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.441385 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.442588 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.442988 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.443123 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:54.447660 (+ 4537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:49643: BlockingRecv error: recv got EOF from 127.27.1.193:49643 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 07 18:28:54 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:54 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:54.901170 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:54.892757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:50317)
0807 18:28:54.893317 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:54.893334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:54.893510 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:54.893819 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:54.893822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:54.894213 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:54.894360 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:54.894369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.895701 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.895707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:54.896699 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:54.896712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.897352 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.897356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:54.897462 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:54.898037 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:54.898055 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:54.900908 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":327,"threads_started":1}
W20260807 18:28:54.901338 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:54.892857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:50317 (local address 127.27.1.254:44139)
0807 18:28:54.893059 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:54.893067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:54.893408 (+ 341us) server_negotiation.cc:407] Connection header received
0807 18:28:54.893615 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:54.893619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:54.893680 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:54.893760 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:54.894479 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.895546 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.896825 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:54.897227 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:54.897319 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:54.901107 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:50317: BlockingRecv error: recv got EOF from 127.27.1.194:50317 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 07 18:28:55 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:55 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:55.377117 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:55.362058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:56855)
0807 18:28:55.362561 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:55.362578 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:55.362688 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:55.363086 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:55.363089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:55.363566 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:55.363711 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:55.363719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.364798 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.364801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:55.365694 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:55.365705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.366405 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.366409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:55.366519 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:55.373885 (+ 7366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:55.373904 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:55.376870 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":278,"threads_started":1}
W20260807 18:28:55.377336 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:55.362241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:56855 (local address 127.27.1.254:44139)
0807 18:28:55.362461 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:55.362467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:55.362649 (+ 182us) server_negotiation.cc:407] Connection header received
0807 18:28:55.362778 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:55.362872 (+ 94us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:55.362953 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:55.363039 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:55.363878 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.364665 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.365837 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.366267 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.366374 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:55.377064 (+ 10690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:56855: BlockingRecv error: recv got EOF from 127.27.1.195:56855 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 07 18:28:55 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:55 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:55.457893 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:55.448846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:55467)
0807 18:28:55.449364 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:55.449380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:55.449583 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:55.449932 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:55.449937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:55.450328 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:55.450534 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:55.450554 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.452026 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.452031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:55.453276 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:55.453289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.453925 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.453929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:55.454052 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:55.454619 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:55.454637 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:55.457524 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":276,"threads_started":1}
W20260807 18:28:55.457953 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:55.448938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:55467 (local address 127.27.1.254:44139)
0807 18:28:55.449178 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:55.449188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:55.449476 (+ 288us) server_negotiation.cc:407] Connection header received
0807 18:28:55.449727 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:55.449732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:55.449793 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:55.449900 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:55.450702 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.451844 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.453400 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.453794 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.453891 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:55.457700 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:55467: BlockingRecv error: recv got EOF from 127.27.1.193:55467 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 07 18:28:55 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:55 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:55.910812 5125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:55.902271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:36231)
0807 18:28:55.902788 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:55.902809 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:55.903003 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:55.903472 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:55.903477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:55.903876 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:55.904053 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:55.904062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.905159 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.905164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:55.906058 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:55.906069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.906769 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.906775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:55.906956 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:55.907613 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:55.907632 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:55.910454 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":287,"threads_started":1}
W20260807 18:28:55.910981 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:55.902315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:36231 (local address 127.27.1.254:44139)
0807 18:28:55.902576 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:55.902583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:55.902920 (+ 337us) server_negotiation.cc:407] Connection header received
0807 18:28:55.903214 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:55.903219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:55.903286 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:55.903404 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:55.904207 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.905007 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.906212 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:55.906611 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:55.906768 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:55.910694 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:36231: BlockingRecv error: recv got EOF from 127.27.1.194:36231 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 07 18:28:56 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:56 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:56.386565 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:56.378072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:58257)
0807 18:28:56.378614 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:56.378629 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:56.378789 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:56.379184 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:56.379189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:56.379584 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:56.379745 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:56.379754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.380769 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.380775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:56.381680 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:56.381692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.382386 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.382392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:56.382570 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:56.383332 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:56.383349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:56.386305 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":298,"threads_started":1}
W20260807 18:28:56.386816 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:56.378144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:58257 (local address 127.27.1.254:44139)
0807 18:28:56.378379 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:56.378387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:56.378714 (+ 327us) server_negotiation.cc:407] Connection header received
0807 18:28:56.378912 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:56.378917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:56.379022 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:56.379125 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:56.379900 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.380627 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.381824 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.382236 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.382362 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:56.386558 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:58257: BlockingRecv error: recv got EOF from 127.27.1.195:58257 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 18:28:56 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:56 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:56.467216 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:56.458925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:34803)
0807 18:28:56.459505 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:56.459521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:56.459674 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:56.460021 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:56.460026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:56.460416 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:56.460573 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:56.460581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.461567 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.461573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:56.462580 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:56.462591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.463328 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.463334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:56.463534 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:56.464115 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:56.464132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:56.466909 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":310,"threads_started":1}
W20260807 18:28:56.467420 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:56.459073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:34803 (local address 127.27.1.254:44139)
0807 18:28:56.459301 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:56.459307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:56.459606 (+ 299us) server_negotiation.cc:407] Connection header received
0807 18:28:56.459815 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:56.459819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:56.459877 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:56.459973 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:56.460713 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.461437 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.462718 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.463166 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.463303 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:56.467113 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:34803: BlockingRecv error: recv got EOF from 127.27.1.193:34803 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 07 18:28:56 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:56 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:56.921094 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:56.911805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:33403)
0807 18:28:56.912373 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:56.912390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:56.912569 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:56.913014 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:56.913020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:56.913424 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:56.913581 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:56.913591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.914596 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.914602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:56.915701 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:56.915715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.916459 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.916465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:56.916652 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:56.917329 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:56.917347 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:56.920841 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":297,"threads_started":1}
W20260807 18:28:56.921277 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:56.911875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:33403 (local address 127.27.1.254:44139)
0807 18:28:56.912151 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:56.912159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:56.912489 (+ 330us) server_negotiation.cc:407] Connection header received
0807 18:28:56.912701 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:56.912708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:56.912841 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:56.912959 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:56.913733 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.914456 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.915854 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:56.916288 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:56.916446 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:56.921034 (+ 4588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:33403: BlockingRecv error: recv got EOF from 127.27.1.194:33403 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 07 18:28:57 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:57 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:57.396842 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:57.387629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:53143)
0807 18:28:57.388170 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:57.388185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:57.388297 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:57.388667 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:57.388670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:57.389024 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:57.389187 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:57.389195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.390608 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.390611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:57.391663 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:57.391675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.392409 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.392413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:57.392549 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:57.393169 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:57.393185 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:57.396520 (+ 3335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":279,"threads_started":1}
W20260807 18:28:57.396973 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:57.387687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:53143 (local address 127.27.1.254:44139)
0807 18:28:57.387864 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:57.387870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:57.388273 (+ 403us) server_negotiation.cc:407] Connection header received
0807 18:28:57.388400 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:57.388406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:57.388512 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:57.388620 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:57.389335 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.390483 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.391789 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.392263 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.392396 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:57.396724 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:53143: BlockingRecv error: recv got EOF from 127.27.1.195:53143 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 07 18:28:57 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:57 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:57.476280 5133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:57.468162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:53195)
0807 18:28:57.468723 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:57.468739 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:57.468845 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:57.469131 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:57.469134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:57.469574 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:57.469771 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:57.469779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.470769 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.470772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:57.471714 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:57.471725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.472416 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.472420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:57.472547 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:57.473123 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:57.473141 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:57.476034 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":287,"threads_started":1}
W20260807 18:28:57.476575 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:57.468232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:53195 (local address 127.27.1.254:44139)
0807 18:28:57.468390 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:57.468395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:57.468818 (+ 423us) server_negotiation.cc:407] Connection header received
0807 18:28:57.468946 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:57.468949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:57.469010 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:57.469091 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:57.469927 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.470642 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.471839 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.472286 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.472381 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:57.476307 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:53195: BlockingRecv error: recv got EOF from 127.27.1.193:53195 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 18:28:57 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:57 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:57.931111 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:57.922292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:56165)
0807 18:28:57.922896 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:57.922916 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:57.923056 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:57.923471 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:57.923476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:57.924247 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:57.924498 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:57.924514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.925556 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.925564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:57.926494 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:57.926506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.927270 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.927276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:57.927450 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:57.928019 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:57.928037 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:57.930746 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":316,"threads_started":1}
W20260807 18:28:57.931244 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:57.922418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:56165 (local address 127.27.1.254:44139)
0807 18:28:57.922626 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:57.922635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:57.922999 (+ 364us) server_negotiation.cc:407] Connection header received
0807 18:28:57.923229 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:57.923235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:57.923321 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:57.923426 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:57.924662 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.925413 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.926644 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:57.927079 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:57.927259 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:57.930961 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:56165: BlockingRecv error: recv got EOF from 127.27.1.194:56165 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 07 18:28:58 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:58 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:58.405925 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:58.397845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:43933)
0807 18:28:58.398410 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:58.398426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:58.398551 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:58.398876 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:58.398880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:58.399356 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:58.399544 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:58.399553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.400511 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.400515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:58.401418 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:58.401428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.402076 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.402080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:58.402197 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:58.402821 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:58.402838 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:58.405687 (+ 2849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":302,"threads_started":1}
W20260807 18:28:58.406215 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:58.397959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:43933 (local address 127.27.1.254:44139)
0807 18:28:58.398156 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:58.398160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:58.398504 (+ 344us) server_negotiation.cc:407] Connection header received
0807 18:28:58.398654 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:58.398657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:58.398735 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:58.398824 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:58.399673 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.400376 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.401549 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.401957 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.402049 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:58.405936 (+ 3887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:43933: BlockingRecv error: recv got EOF from 127.27.1.195:43933 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 18:28:58 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:58 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:58.486963 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:58.477352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:53601)
0807 18:28:58.478064 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:58.478079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:58.478236 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:58.478735 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:58.478740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:58.479317 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:58.479554 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:58.479568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.480761 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.480765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:58.481890 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:58.481902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.483037 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.483041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:58.483242 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:58.483835 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:58.483853 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:58.486670 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":336,"threads_started":1}
W20260807 18:28:58.487191 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:58.477537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:53601 (local address 127.27.1.254:44139)
0807 18:28:58.477745 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:58.477751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:58.478173 (+ 422us) server_negotiation.cc:407] Connection header received
0807 18:28:58.478451 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:58.478457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:58.478538 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:58.478628 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:58.479700 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.480635 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.482116 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.482885 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.482992 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:58.486912 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:53601: BlockingRecv error: recv got EOF from 127.27.1.193:53601 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 07 18:28:58 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:58 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:58.940065 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:58.932199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:42495)
0807 18:28:58.932768 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:58.932784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:58.932897 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:58.933209 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:58.933212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:58.933626 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:58.933785 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:58.933793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.934781 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.934785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:58.935760 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:58.935773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.936418 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.936423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:58.936552 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:58.937181 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:58.937198 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:58.939845 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":331,"threads_started":1}
W20260807 18:28:58.940378 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:58.932301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:42495 (local address 127.27.1.254:44139)
0807 18:28:58.932514 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:58.932520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:58.932857 (+ 337us) server_negotiation.cc:407] Connection header received
0807 18:28:58.933017 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:58.933020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:58.933082 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:58.933165 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:58.933927 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.934659 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.935886 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:58.936283 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:58.936382 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:58.940090 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:42495: BlockingRecv error: recv got EOF from 127.27.1.194:42495 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 07 18:28:59 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:59 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:59.415462 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:59.407138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:33881)
0807 18:28:59.407819 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:59.407841 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:59.407978 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:59.408333 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:59.408337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:59.408917 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:59.409074 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:59.409083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.410121 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.410124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:59.411031 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:59.411041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.411819 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.411823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:59.411945 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:59.412553 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:59.412570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:59.415248 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":351,"threads_started":1}
W20260807 18:28:59.415704 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:59.407259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:33881 (local address 127.27.1.254:44139)
0807 18:28:59.407502 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:59.407512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:59.407939 (+ 427us) server_negotiation.cc:407] Connection header received
0807 18:28:59.408081 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:59.408086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:59.408175 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:59.408281 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:59.409238 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.409994 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.411199 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.411661 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.411768 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:59.415440 (+ 3672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:33881: BlockingRecv error: recv got EOF from 127.27.1.195:33881 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 18:28:59 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:59 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:59.496838 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:59.487899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:44983)
0807 18:28:59.488480 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:59.488503 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:59.488630 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:59.488921 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:59.488924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:59.489330 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:59.489512 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:59.489521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.490817 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.490821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:59.492267 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:59.492281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.492990 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.493012 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:59.493170 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:59.493759 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:59.493777 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:59.496592 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":272,"threads_started":1}
W20260807 18:28:59.496992 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:59.487999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:44983 (local address 127.27.1.254:44139)
0807 18:28:59.488180 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:59.488185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:59.488576 (+ 391us) server_negotiation.cc:407] Connection header received
0807 18:28:59.488730 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:59.488733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:59.488795 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:59.488879 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:59.489634 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.490668 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.492406 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.492846 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.492963 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:59.496778 (+ 3815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:44983: BlockingRecv error: recv got EOF from 127.27.1.193:44983 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 18:28:59 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:28:59 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:28:59.950769 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:28:59.941105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:34593)
0807 18:28:59.941746 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0807 18:28:59.941791 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 18:28:59.941942 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:28:59.942444 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:28:59.942450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:28:59.943020 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:28:59.943249 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:59.943262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.944456 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.944463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:59.945970 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:28:59.945984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.946742 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.946749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:28:59.946927 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:59.947482 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:28:59.947499 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:28:59.950401 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":283,"threads_started":1}
W20260807 18:28:59.950845 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:28:59.941153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:34593 (local address 127.27.1.254:44139)
0807 18:28:59.941485 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0807 18:28:59.941497 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:28:59.941907 (+ 410us) server_negotiation.cc:407] Connection header received
0807 18:28:59.942131 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:28:59.942137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:28:59.942226 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:28:59.942346 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:28:59.943426 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.944294 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.946134 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:28:59.946572 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:28:59.946709 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:28:59.950595 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:34593: BlockingRecv error: recv got EOF from 127.27.1.194:34593 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 07 18:29:00 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:00 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:00.431005 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:00.416437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:43325)
0807 18:29:00.417081 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:00.417099 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:00.417232 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:00.417546 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:00.417549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:00.417958 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:00.418127 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:00.418135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.419206 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.419211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:00.420458 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:00.420471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.421380 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.421384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:00.421511 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:00.427580 (+ 6069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:00.427599 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:00.430635 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":297,"threads_started":1}
W20260807 18:29:00.431077 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:00.416613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:43325 (local address 127.27.1.254:44139)
0807 18:29:00.416813 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:00.416817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:00.417188 (+ 371us) server_negotiation.cc:407] Connection header received
0807 18:29:00.417336 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:00.417340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:00.417402 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:00.417487 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:00.418252 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.418949 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.420608 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.421240 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.421338 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:00.430847 (+ 9509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:43325: BlockingRecv error: recv got EOF from 127.27.1.195:43325 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 07 18:29:00 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:00 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:00.506593 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:00.497903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:46229)
0807 18:29:00.498507 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:00.498524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:00.498639 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:00.498933 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:00.498936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:00.499367 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:00.499537 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:00.499545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.500519 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.500523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:00.501426 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:00.501436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.502179 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.502186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:00.502336 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:00.503072 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:00.503095 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:00.506213 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":337,"threads_started":1}
W20260807 18:29:00.506685 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:00.498079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:46229 (local address 127.27.1.254:44139)
0807 18:29:00.498310 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:00.498315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:00.498595 (+ 280us) server_negotiation.cc:407] Connection header received
0807 18:29:00.498734 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:00.498738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:00.498799 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:00.498883 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:00.499659 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.500382 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.501551 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.501992 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.502083 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:00.506418 (+ 4335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:46229: BlockingRecv error: recv got EOF from 127.27.1.193:46229 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 07 18:29:00 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:00 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:00.960480 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:00.951906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:55773)
0807 18:29:00.952549 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:00.952571 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:00.952733 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:00.953151 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:00.953156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:00.953594 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:00.953741 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:00.953750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.954935 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.954940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:00.955862 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:00.955873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.956619 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.956639 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:00.956798 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:00.957405 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:00.957423 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:00.960243 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":332,"threads_started":1}
W20260807 18:29:00.960680 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:00.951983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:55773 (local address 127.27.1.254:44139)
0807 18:29:00.952227 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:00.952235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:00.952658 (+ 423us) server_negotiation.cc:407] Connection header received
0807 18:29:00.952907 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:00.952912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:00.953004 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:00.953116 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:00.953894 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.954782 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.956005 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:00.956432 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:00.956583 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:00.960425 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:55773: BlockingRecv error: recv got EOF from 127.27.1.194:55773 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 07 18:29:01 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:01 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:01.440682 5145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:01.432185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:55453)
0807 18:29:01.432781 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:01.432797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:01.432965 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:01.433312 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:01.433317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:01.433806 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:01.433956 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:01.433964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.435029 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.435036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:01.435946 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:01.435958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.436738 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.436745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:01.436909 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:01.437580 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:01.437598 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:01.440417 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":315,"threads_started":1}
W20260807 18:29:01.440868 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:01.432296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:55453 (local address 127.27.1.254:44139)
0807 18:29:01.432509 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:01.432518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:01.432888 (+ 370us) server_negotiation.cc:407] Connection header received
0807 18:29:01.433100 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:01.433109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:01.433191 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:01.433282 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:01.434120 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.434859 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.436103 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.436539 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.436676 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:01.440617 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:55453: BlockingRecv error: recv got EOF from 127.27.1.195:55453 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 07 18:29:01 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:01 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:01.516294 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:01.507709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:58727)
0807 18:29:01.508276 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:01.508291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:01.508479 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:01.508837 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:01.508842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:01.509270 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:01.509437 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:01.509446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.510512 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.510517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:01.511501 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:01.511513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.512228 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.512234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:01.512438 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:01.513083 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:01.513102 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:01.515935 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":320,"threads_started":1}
W20260807 18:29:01.516393 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:01.507782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:58727 (local address 127.27.1.254:44139)
0807 18:29:01.508037 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:01.508046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:01.508388 (+ 342us) server_negotiation.cc:407] Connection header received
0807 18:29:01.508617 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:01.508625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:01.508692 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:01.508800 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:01.509610 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.510365 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.511652 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.512065 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.512236 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:01.516154 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:58727: BlockingRecv error: recv got EOF from 127.27.1.193:58727 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 07 18:29:01 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:01 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:01.969846 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:01.961480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:41651)
0807 18:29:01.962095 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:01.962115 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:01.962257 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:01.962685 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:01.962693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:01.963291 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:01.963528 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:01.963543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.964568 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.964571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:01.965490 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:01.965502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.966134 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.966138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:01.966249 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:01.966816 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:01.966834 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:01.969601 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":314,"threads_started":1}
W20260807 18:29:01.970052 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:01.961564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:41651 (local address 127.27.1.254:44139)
0807 18:29:01.961771 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:01.961814 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:01.962215 (+ 401us) server_negotiation.cc:407] Connection header received
0807 18:29:01.962376 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:01.962384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:01.962464 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:01.962611 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:01.963696 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.964442 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.965612 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:01.966013 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:01.966101 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:01.969802 (+ 3701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:41651: BlockingRecv error: recv got EOF from 127.27.1.194:41651 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 07 18:29:02 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:02 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:02.449954 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:02.441563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:44731)
0807 18:29:02.442065 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:02.442082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:02.442196 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:02.442499 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:02.442502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:02.442870 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:02.443050 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:02.443059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.444095 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.444099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:02.445032 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:02.445042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.445677 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.445681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:02.445800 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:02.446364 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:02.446381 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:02.449641 (+ 3260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":263,"threads_started":1}
W20260807 18:29:02.450033 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:02.441637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:44731 (local address 127.27.1.254:44139)
0807 18:29:02.441877 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:02.441883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:02.442164 (+ 281us) server_negotiation.cc:407] Connection header received
0807 18:29:02.442297 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:02.442301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:02.442363 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:02.442446 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:02.443203 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.443980 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.445150 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.445549 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.445638 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:02.449848 (+ 4210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:44731: BlockingRecv error: recv got EOF from 127.27.1.195:44731 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 07 18:29:02 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:02 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:02.525789 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:02.517196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:37715)
0807 18:29:02.517761 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:02.517776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:02.517888 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:02.518199 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:02.518203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:02.518588 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:02.518765 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:02.518777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.520053 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.520059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:02.521170 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:02.521182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.522052 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.522056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:02.522180 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:02.522817 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:02.522836 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:02.525560 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":342,"threads_started":1}
W20260807 18:29:02.525981 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:02.517387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:37715 (local address 127.27.1.254:44139)
0807 18:29:02.517574 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:02.517578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:02.517852 (+ 274us) server_negotiation.cc:407] Connection header received
0807 18:29:02.518005 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:02.518009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:02.518071 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:02.518153 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:02.518923 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.519897 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.521299 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.521893 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.522008 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:02.525724 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:37715: BlockingRecv error: recv got EOF from 127.27.1.193:37715 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 18:29:02 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:02 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:02.979697 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:02.970790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:36407)
0807 18:29:02.971362 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:02.971378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:02.971502 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:02.971799 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:02.971802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:02.972165 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:02.972324 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:02.972332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.973490 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.973494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:02.974402 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:02.974413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.975038 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.975042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:02.975234 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:02.976010 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:02.976033 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:02.979352 (+ 3319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":350,"threads_started":1}
W20260807 18:29:02.979826 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:02.970888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:36407 (local address 127.27.1.254:44139)
0807 18:29:02.971074 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:02.971078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:02.971455 (+ 377us) server_negotiation.cc:407] Connection header received
0807 18:29:02.971603 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:02.971606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:02.971667 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:02.971748 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:02.972474 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.973356 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.974531 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:02.974913 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:02.975010 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:02.979561 (+ 4551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:36407: BlockingRecv error: recv got EOF from 127.27.1.194:36407 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 18:29:03 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:03 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:03.459419 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:03.451053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:50497)
0807 18:29:03.451663 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:03.451679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:03.451785 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:03.452116 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:03.452120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:03.452479 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:03.452675 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:03.452690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.453668 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.453672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:03.454577 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:03.454588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.455251 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.455255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:03.455378 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:03.455957 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:03.455973 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:03.459104 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":363,"threads_started":1}
W20260807 18:29:03.459571 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:03.451220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:50497 (local address 127.27.1.254:44139)
0807 18:29:03.451403 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:03.451408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:03.451749 (+ 341us) server_negotiation.cc:407] Connection header received
0807 18:29:03.451924 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:03.451928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:03.451989 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:03.452074 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:03.452828 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.453546 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.454699 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.455085 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.455198 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:03.459352 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:50497: BlockingRecv error: recv got EOF from 127.27.1.195:50497 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 18:29:03 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:03 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:03.535943 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:03.526789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:38037)
0807 18:29:03.527367 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:03.527382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:03.527555 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:03.527928 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:03.527933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:03.528521 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:03.528769 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:03.528780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.529973 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.529978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:03.531061 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:03.531075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.531923 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.531928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:03.532045 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:03.532639 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:03.532666 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:03.535694 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":319,"threads_started":1}
W20260807 18:29:03.536132 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:03.526894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:38037 (local address 127.27.1.254:44139)
0807 18:29:03.527183 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:03.527190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:03.527471 (+ 281us) server_negotiation.cc:407] Connection header received
0807 18:29:03.527722 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:03.527727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:03.527789 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:03.527897 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:03.528923 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.529823 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.531222 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.531743 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.531841 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:03.535909 (+ 4068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:38037: BlockingRecv error: recv got EOF from 127.27.1.193:38037 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 07 18:29:03 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:03 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:03.990087 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:03.980654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:55837)
0807 18:29:03.981130 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:03.981148 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:03.981268 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:03.981584 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:03.981587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:03.981996 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:03.982177 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:03.982185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.983268 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.983272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:03.984300 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:03.984314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.985225 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.985232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:03.985398 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:03.986175 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:03.986202 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:03.989826 (+ 3624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
W20260807 18:29:03.990298 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:03.980759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:55837 (local address 127.27.1.254:44139)
0807 18:29:03.981011 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:03.981017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:03.981235 (+ 218us) server_negotiation.cc:407] Connection header received
0807 18:29:03.981376 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:03.981379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:03.981442 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:03.981538 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:03.982306 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.983101 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.984430 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:03.985058 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:03.985158 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:03.990055 (+ 4897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:55837: BlockingRecv error: recv got EOF from 127.27.1.194:55837 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 07 18:29:04 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:04 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:04.469085 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:04.460412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:56731)
0807 18:29:04.460900 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:04.460917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:04.461041 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:04.461337 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:04.461340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:04.461757 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:04.461968 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:04.461980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.463531 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.463536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:04.464510 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:04.464524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.465182 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.465187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:04.465331 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:04.465979 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:04.465998 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:04.468843 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":265,"threads_started":1}
W20260807 18:29:04.469283 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:04.460569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:56731 (local address 127.27.1.254:44139)
0807 18:29:04.460799 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:04.460805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:04.461003 (+ 198us) server_negotiation.cc:407] Connection header received
0807 18:29:04.461144 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:04.461147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:04.461211 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:04.461294 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:04.462159 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.463344 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.464647 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.465043 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.465137 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:04.469036 (+ 3899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:56731: BlockingRecv error: recv got EOF from 127.27.1.195:56731 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 07 18:29:04 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:04 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:04.545647 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:04.536896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:43829)
0807 18:29:04.537493 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:04.537510 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:04.537628 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:04.537942 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:04.537945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:04.538338 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:04.538510 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:04.538518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.539636 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.539640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:04.540548 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:04.540559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.541281 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.541285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:04.541417 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:04.542093 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:04.542112 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:04.545298 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":336,"threads_started":1}
W20260807 18:29:04.545744 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:04.536940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:43829 (local address 127.27.1.254:44139)
0807 18:29:04.537109 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:04.537113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:04.537583 (+ 470us) server_negotiation.cc:407] Connection header received
0807 18:29:04.537739 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:04.537742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:04.537803 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:04.537882 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:04.538632 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.539481 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.540679 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.541148 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.541248 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:04.545504 (+ 4256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:43829: BlockingRecv error: recv got EOF from 127.27.1.193:43829 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 07 18:29:04 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:04 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:05.000777 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:04.991215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:42365)
0807 18:29:04.991821 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:04.991837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:04.991997 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:04.992438 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:04.992443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:04.993007 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:04.993266 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:04.993282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.994426 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.994430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:04.995749 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:04.995763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.996745 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.996751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:04.996932 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:04.997636 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:04.997655 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:05.000540 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":319,"threads_started":1}
W20260807 18:29:05.000945 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:04.991239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:42365 (local address 127.27.1.254:44139)
0807 18:29:04.991529 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:04.991538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:04.991926 (+ 388us) server_negotiation.cc:407] Connection header received
0807 18:29:04.992140 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:04.992147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:04.992232 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:04.992337 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:04.993413 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.994283 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.995893 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:04.996584 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:04.996719 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:05.000735 (+ 4016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:42365: BlockingRecv error: recv got EOF from 127.27.1.194:42365 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 07 18:29:05 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:05 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:05.485208 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:05.470073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:44939)
0807 18:29:05.470702 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:05.470718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:05.470872 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:05.471264 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:05.471267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:05.471834 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:05.472054 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:05.472068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:05.473133 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:05.473138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:05.474033 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:05.474044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:05.474708 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:05.474712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:05.474836 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:05.481547 (+ 6711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:05.481565 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:05.484854 (+ 3289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":369,"threads_started":1}
W20260807 18:29:05.485301 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:05.470136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:44939 (local address 127.27.1.254:44139)
0807 18:29:05.470338 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:05.470343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:05.470814 (+ 471us) server_negotiation.cc:407] Connection header received
0807 18:29:05.471010 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:05.471014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:05.471080 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:05.471223 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:05.472211 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:05.473000 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:05.474152 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:05.474586 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:05.474680 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:05.485061 (+ 10381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:44939: BlockingRecv error: recv got EOF from 127.27.1.195:44939 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 07 18:29:05 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:05 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:05.555380 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:05.546635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:40603)
0807 18:29:05.547238 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:05.547255 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:05.547371 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:05.547695 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:05.547698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:05.548097 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:05.548257 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:05.548265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:05.549272 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:05.549276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:05.550306 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:05.550320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:05.551114 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:05.551124 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:05.551298 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:05.551960 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:05.551978 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:05.555002 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":323,"threads_started":1}
W20260807 18:29:05.555573 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:05.546726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:40603 (local address 127.27.1.254:44139)
0807 18:29:05.546889 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:05.546894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:05.547326 (+ 432us) server_negotiation.cc:407] Connection header received
0807 18:29:05.547466 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:05.547469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:05.547560 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:05.547645 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:05.548380 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:05.549107 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:05.550441 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:05.550880 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:05.550970 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:05.555259 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:40603: BlockingRecv error: recv got EOF from 127.27.1.193:40603 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 07 18:29:06 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:06 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:06.010124 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:06.001754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:38145)
0807 18:29:06.002346 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:06.002362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:06.002517 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:06.002866 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:06.002871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:06.003349 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:06.003513 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:06.003522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.004572 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.004578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:06.005497 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:06.005508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.006287 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.006294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:06.006454 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:06.007016 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:06.007036 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:06.009872 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":311,"threads_started":1}
W20260807 18:29:06.010301 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:06.001924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:38145 (local address 127.27.1.254:44139)
0807 18:29:06.002147 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:06.002154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:06.002441 (+ 287us) server_negotiation.cc:407] Connection header received
0807 18:29:06.002643 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:06.002649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:06.002726 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:06.002830 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:06.003654 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.004421 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.005642 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.006108 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.006252 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:06.010091 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:38145: BlockingRecv error: recv got EOF from 127.27.1.194:38145 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 07 18:29:06 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:06 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:06.495007 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:06.486177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:43915)
0807 18:29:06.486774 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:06.486791 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:06.486910 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:06.487231 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:06.487235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:06.487638 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:06.487855 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:06.487867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.488990 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.488994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:06.489903 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:06.489913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.490667 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.490671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:06.490800 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:06.491507 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:06.491525 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:06.494647 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":301,"threads_started":1}
W20260807 18:29:06.495093 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:06.486236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:43915 (local address 127.27.1.254:44139)
0807 18:29:06.486400 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:06.486404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:06.486863 (+ 459us) server_negotiation.cc:407] Connection header received
0807 18:29:06.487012 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:06.487015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:06.487077 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:06.487220 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:06.488002 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.488855 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.490048 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.490504 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.490609 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:06.494852 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:43915: BlockingRecv error: recv got EOF from 127.27.1.195:43915 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 07 18:29:06 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:06 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:06.565418 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:06.556342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:42907)
0807 18:29:06.556892 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:06.556918 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:06.557059 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:06.557391 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:06.557395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:06.557800 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:06.558004 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:06.558014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.559022 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.559026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:06.560121 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:06.560136 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.560825 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.560831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:06.560969 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:06.561716 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:06.561736 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:06.565026 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":312,"threads_started":1}
W20260807 18:29:06.565507 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:06.556436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:42907 (local address 127.27.1.254:44139)
0807 18:29:06.556666 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:06.556671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:06.557010 (+ 339us) server_negotiation.cc:407] Connection header received
0807 18:29:06.557159 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:06.557163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:06.557240 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:06.557321 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:06.558135 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.558873 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.560256 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:06.560678 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:06.560768 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:06.565237 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:42907: BlockingRecv error: recv got EOF from 127.27.1.193:42907 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 07 18:29:07 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:07 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:07.020002 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:07.011208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:55001)
0807 18:29:07.011755 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:07.011775 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:07.011945 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:07.012349 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:07.012354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:07.012804 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:07.012997 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:07.013007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.014141 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.014145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:07.015088 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:07.015101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.015864 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.015868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:07.015999 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:07.016629 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:07.016648 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:07.019729 (+ 3081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":261,"threads_started":1}
W20260807 18:29:07.020145 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:07.011297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:55001 (local address 127.27.1.254:44139)
0807 18:29:07.011497 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:07.011505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:07.011879 (+ 374us) server_negotiation.cc:407] Connection header received
0807 18:29:07.012080 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:07.012085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:07.012174 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:07.012257 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:07.013136 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.013999 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.015269 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.015708 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.015808 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:07.019939 (+ 4131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:55001: BlockingRecv error: recv got EOF from 127.27.1.194:55001 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 18:29:07 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:07 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.195@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:07.504837 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:07.496084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.195:41723)
0807 18:29:07.496587 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:07.496603 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:07.496766 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:07.497101 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:07.497106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:07.497523 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:07.497699 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:07.497707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.498729 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.498735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:07.499673 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:07.499684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.500427 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.500433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:07.500638 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:07.501357 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:07.501380 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:07.504586 (+ 3206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":244,"threads_started":1}
W20260807 18:29:07.505036 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:07.496147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:41723 (local address 127.27.1.254:44139)
0807 18:29:07.496385 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:07.496393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:07.496696 (+ 303us) server_negotiation.cc:407] Connection header received
0807 18:29:07.496901 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:07.496904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:07.496965 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:07.497068 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:07.497848 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.498581 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.499829 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.500274 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.500405 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:07.504800 (+ 4395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.195:41723: BlockingRecv error: recv got EOF from 127.27.1.195:41723 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 18:29:07 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:07 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.193@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:07.575256 5167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:07.566577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.193:39877)
0807 18:29:07.567163 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:07.567179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:07.567297 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:07.567642 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:07.567645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:07.568061 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:07.568251 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:07.568259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.569219 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.569222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:07.570198 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:07.570209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.570881 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.570885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:07.571017 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:07.571616 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:07.571633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:07.574915 (+ 3282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":311,"threads_started":1}
W20260807 18:29:07.575394 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:07.566666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:39877 (local address 127.27.1.254:44139)
0807 18:29:07.566862 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:07.566867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:07.567258 (+ 391us) server_negotiation.cc:407] Connection header received
0807 18:29:07.567400 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:07.567404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:07.567510 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:07.567597 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:07.568389 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.569099 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.570329 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:07.570758 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:07.570851 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:07.575106 (+ 4255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.193:39877: BlockingRecv error: recv got EOF from 127.27.1.193:39877 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 07 18:29:08 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
Aug 07 18:29:08 dist-test-slave-1xrh krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.1.194@KRBTEST.COM for oryx/127.27.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260807 18:29:08.029866 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:08.021290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44139 (local address 127.27.1.194:37391)
0807 18:29:08.021889 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:08.021905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:08.022057 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:08.022445 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:08.022449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:08.023039 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:08.023331 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:08.023343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:08.024338 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:08.024342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:08.025305 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:08.025316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:08.026018 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:08.026023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:08.026144 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:08.026730 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:08.026747 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:08.029614 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.1.254:44139: Server oryx/127.27.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":349,"threads_started":1}
W20260807 18:29:08.030036 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:08.021385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:37391 (local address 127.27.1.254:44139)
0807 18:29:08.021574 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:08.021578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:08.022002 (+ 424us) server_negotiation.cc:407] Connection header received
0807 18:29:08.022196 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:08.022200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:08.022284 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:08.022386 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:08.023471 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:08.024206 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:08.025430 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:08.025882 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:08.025982 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:08.029803 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.1.194:37391: BlockingRecv error: recv got EOF from 127.27.1.194:37391 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 18:29:08.330703 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4651
I20260807 18:29:08.343104 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:08.331879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.27.1.193:34869)
0807 18:29:08.332537 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:08.332544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:08.332571 (+ 27us) server_negotiation.cc:407] Connection header received
0807 18:29:08.332632 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:08.332638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:08.332891 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:08.333039 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:08.334565 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:08.335745 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:08.336931 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:08.337490 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:08.337596 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:08.340067 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:08.340075 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:08.340089 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:08.340123 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:08.342617 (+ 2494us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.1.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 18:29:08.342884 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56058: Request ticket server kudu/127.27.1.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":353,"threads_started":1}
W20260807 18:29:08.343302 5170 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56058: Request ticket server kudu/127.27.1.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 18:29:08.343305 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4651: Not authorized: Client connection negotiation failed: client connection to 127.27.1.193:34869: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.1.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 18:29:08.343497 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 4651
I20260807 18:29:08.363397 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4819
I20260807 18:29:08.375268 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:08.364434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.27.1.194:43951)
0807 18:29:08.365059 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:08.365067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:08.365097 (+ 30us) server_negotiation.cc:407] Connection header received
0807 18:29:08.365157 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:08.365163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:08.365386 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:08.365544 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:08.366820 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:08.368008 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:08.369184 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:08.369719 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:08.369816 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:08.372339 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:08.372348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:08.372361 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:08.372395 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:08.374683 (+ 2288us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.1.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 18:29:08.374933 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58564: Request ticket server kudu/127.27.1.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":295,"threads_started":1}
W20260807 18:29:08.375352 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4819: Not authorized: Client connection negotiation failed: client connection to 127.27.1.194:43951: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.1.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 18:29:08.375384 5171 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58564: Request ticket server kudu/127.27.1.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 18:29:08.375526 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 4819
I20260807 18:29:08.396059 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4964
I20260807 18:29:08.408113 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:08.397100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.27.1.195:41209)
0807 18:29:08.397663 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:08.397671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:08.397699 (+ 28us) server_negotiation.cc:407] Connection header received
0807 18:29:08.397760 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:08.397767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:08.398031 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:08.398186 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:08.399611 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:08.400631 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:08.401847 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:08.402376 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:08.402483 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:08.405069 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:08.405077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:08.405090 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:08.405127 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:08.407590 (+ 2463us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.1.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 18:29:08.407836 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37482: Request ticket server kudu/127.27.1.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":303,"threads_started":1}
W20260807 18:29:08.408221 5173 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37482: Request ticket server kudu/127.27.1.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 18:29:08.408233 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4964: Not authorized: Client connection negotiation failed: client connection to 127.27.1.195:41209: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.1.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 18:29:08.408447 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 4964
I20260807 18:29:08.428689 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 4579
I20260807 18:29:08.429916 4644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:08.548655 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 4579
2026-08-07T18:29:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21277 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T18:29:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:29:08Z Disabled control of system clock
I20260807 18:29:08.637262 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38917
--webserver_interface=127.27.1.254
--webserver_port=0
--builtin_ntp_servers=127.27.1.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:38917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 18:29:08.966073 5179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:08.966353 5179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:08.966434 5179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:08.973667 5179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:29:08.973769 5179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:08.973836 5179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:08.973924 5179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:29:08.974010 5179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:29:08.983058 5179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:38917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:08.984408 5179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:08.986182 5179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:08.998119 5184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:08.998154 5187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:08.998222 5185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:09.000663 5179 server_base.cc:1034] running on GCE node
I20260807 18:29:09.001353 5179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:09.002614 5179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:09.003818 5179 hybrid_clock.cc:648] HybridClock initialized: now 1786127349003767 us; error 48 us; skew 500 ppm
I20260807 18:29:09.004181 5179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:29:09.007104 5179 webserver.cc:460] Webserver started at http://127.27.1.254:44395/ using document root <none> and password file <none>
I20260807 18:29:09.008062 5179 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:09.008208 5179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:09.008548 5179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:09.011539 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "c180ef19816a419abfa11ef2fbe68173"
format_stamp: "Formatted at 2026-08-07 18:29:09 on dist-test-slave-1xrh"
server_key: "6a1c69e73f5673a68d0727e68ad73b5c"
server_key_iv: "bbca9212d0bc6089bd5b44c9f290a0d2"
server_key_version: "encryptionkey@0"
I20260807 18:29:09.012319 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "c180ef19816a419abfa11ef2fbe68173"
format_stamp: "Formatted at 2026-08-07 18:29:09 on dist-test-slave-1xrh"
server_key: "6a1c69e73f5673a68d0727e68ad73b5c"
server_key_iv: "bbca9212d0bc6089bd5b44c9f290a0d2"
server_key_version: "encryptionkey@0"
I20260807 18:29:09.017472 5179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:29:09.021001 5194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:09.022329 5179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 18:29:09.022518 5179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "c180ef19816a419abfa11ef2fbe68173"
format_stamp: "Formatted at 2026-08-07 18:29:09 on dist-test-slave-1xrh"
server_key: "6a1c69e73f5673a68d0727e68ad73b5c"
server_key_iv: "bbca9212d0bc6089bd5b44c9f290a0d2"
server_key_version: "encryptionkey@0"
I20260807 18:29:09.022737 5179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:09.056094 5179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:09.056962 5179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:09.057274 5179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:09.076009 5179 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:38917
I20260807 18:29:09.076040 5251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:38917 every 8 connection(s)
I20260807 18:29:09.077311 5179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:29:09.080295 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 5179
I20260807 18:29:09.080504 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:29:09.081076 5253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:29:09.081233 27655 external_mini_cluster.cc:1428] Setting key 403643cd157c598ca72d0dcca0fd1176
I20260807 18:29:09.094612 5253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173: Bootstrap starting.
I20260807 18:29:09.094991 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:09.083313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.27.1.254:38917)
0807 18:29:09.084324 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:09.084343 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:09.084383 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:29:09.085882 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:09.085904 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:09.086376 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:09.086713 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:09.087455 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:09.088721 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:09.090592 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:09.091311 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:09.091444 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:09.092721 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:09.092727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:09.092742 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:09.093250 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:09.093324 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:09.093344 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:09.093507 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:09.093730 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 18:29:09.094041 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":403,"threads_started":1}
I20260807 18:29:09.098116 5253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173: Neither blocks nor log segments found. Creating new log.
I20260807 18:29:09.099071 5253 log.cc:824] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173: Log is configured to *not* fsync() on all Append() calls
I20260807 18:29:09.101783 5253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173: No bootstrap required, opened a new log
I20260807 18:29:09.109027 5253 raft_consensus.cc:348] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c180ef19816a419abfa11ef2fbe68173" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38917 } }
I20260807 18:29:09.109390 5253 raft_consensus.cc:374] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:29:09.109495 5253 raft_consensus.cc:729] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c180ef19816a419abfa11ef2fbe68173, State: Initialized, Role: FOLLOWER
I20260807 18:29:09.110090 5253 consensus_queue.cc:260] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c180ef19816a419abfa11ef2fbe68173" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38917 } }
I20260807 18:29:09.110325 5253 raft_consensus.cc:388] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:29:09.110471 5253 raft_consensus.cc:482] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:29:09.110654 5253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:29:09.112944 5253 raft_consensus.cc:504] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c180ef19816a419abfa11ef2fbe68173" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38917 } }
I20260807 18:29:09.113385 5253 leader_election.cc:302] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c180ef19816a419abfa11ef2fbe68173; no voters:
I20260807 18:29:09.113895 5253 leader_election.cc:288] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:29:09.114015 5259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:29:09.114745 5259 raft_consensus.cc:686] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [term 1 LEADER]: Becoming Leader. State: Replica: c180ef19816a419abfa11ef2fbe68173, State: Running, Role: LEADER
I20260807 18:29:09.115335 5259 consensus_queue.cc:237] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c180ef19816a419abfa11ef2fbe68173" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38917 } }
I20260807 18:29:09.115940 5253 sys_catalog.cc:564] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:29:09.118206 5261 sys_catalog.cc:455] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c180ef19816a419abfa11ef2fbe68173. Latest consensus state: current_term: 1 leader_uuid: "c180ef19816a419abfa11ef2fbe68173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c180ef19816a419abfa11ef2fbe68173" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38917 } } }
I20260807 18:29:09.118708 5261 sys_catalog.cc:458] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [sys.catalog]: This master's current role is: LEADER
I20260807 18:29:09.118299 5260 sys_catalog.cc:455] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c180ef19816a419abfa11ef2fbe68173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c180ef19816a419abfa11ef2fbe68173" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38917 } } }
I20260807 18:29:09.121718 5260 sys_catalog.cc:458] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173 [sys.catalog]: This master's current role is: LEADER
I20260807 18:29:09.122364 5266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:29:09.127247 5266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:29:09.135286 5266 catalog_manager.cc:1347] Generated new cluster ID: 6f56221618b0477e8883000a488c5e4e
I20260807 18:29:09.135398 5266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:29:09.155810 5266 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:29:09.157037 5266 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:29:09.166565 5266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c180ef19816a419abfa11ef2fbe68173: Generated new TSK 0
I20260807 18:29:09.167411 5266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:29:09.178049 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38917
--builtin_ntp_servers=127.27.1.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 18:29:09.529358 5280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:09.529659 5280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:09.529838 5280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:09.537876 5280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:09.537967 5280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:09.538117 5280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:29:09.547619 5280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:09.548928 5280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:09.550850 5280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:09.564200 5286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:09.564193 5285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:09.564191 5288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:09.565060 5280 server_base.cc:1034] running on GCE node
I20260807 18:29:09.565603 5280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:09.566510 5280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:09.567760 5280 hybrid_clock.cc:648] HybridClock initialized: now 1786127349567713 us; error 50 us; skew 500 ppm
I20260807 18:29:09.568085 5280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:29:09.570381 5280 webserver.cc:460] Webserver started at http://127.27.1.193:44603/ using document root <none> and password file <none>
I20260807 18:29:09.571080 5280 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:09.571242 5280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:09.571591 5280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:09.574340 5280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "a7a8dc2f19704f2a92a1223a5dcaa246"
format_stamp: "Formatted at 2026-08-07 18:29:09 on dist-test-slave-1xrh"
server_key: "5a5e5ac8c8889aba8a8b03bd1e413155"
server_key_iv: "d68178030d64bfb20e88bac511922523"
server_key_version: "encryptionkey@0"
I20260807 18:29:09.575235 5280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "a7a8dc2f19704f2a92a1223a5dcaa246"
format_stamp: "Formatted at 2026-08-07 18:29:09 on dist-test-slave-1xrh"
server_key: "5a5e5ac8c8889aba8a8b03bd1e413155"
server_key_iv: "d68178030d64bfb20e88bac511922523"
server_key_version: "encryptionkey@0"
I20260807 18:29:09.580312 5280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 18:29:09.583678 5294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:09.584936 5280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:29:09.585129 5280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "a7a8dc2f19704f2a92a1223a5dcaa246"
format_stamp: "Formatted at 2026-08-07 18:29:09 on dist-test-slave-1xrh"
server_key: "5a5e5ac8c8889aba8a8b03bd1e413155"
server_key_iv: "d68178030d64bfb20e88bac511922523"
server_key_version: "encryptionkey@0"
I20260807 18:29:09.585345 5280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:09.600034 5280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:09.600948 5280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:09.601253 5280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:09.602151 5280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:29:09.603612 5280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:29:09.603737 5280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:09.603880 5280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:29:09.603978 5280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:09.637691 5280 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:35335
I20260807 18:29:09.637764 5415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:35335 every 8 connection(s)
I20260807 18:29:09.639034 5280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:29:09.648350 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 5280
I20260807 18:29:09.648564 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:29:09.649296 27655 external_mini_cluster.cc:1428] Setting key 707470e2e2a2b090a0a12997346b1b7f
I20260807 18:29:09.651590 5418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:09.641639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:53157 (local address 127.27.1.254:38917)
0807 18:29:09.642054 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:09.642058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:09.643371 (+ 1313us) server_negotiation.cc:407] Connection header received
0807 18:29:09.645046 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:09.645050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:09.645117 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:09.645216 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:09.646176 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:09.646894 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:09.648550 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:09.649002 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:09.649100 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:09.650638 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:09.650644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:09.650648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:09.650666 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:09.650711 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:09.650726 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:09.650816 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:09.651173 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 18:29:09.651273 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":197,"threads_started":1}
I20260807 18:29:09.652352 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38917
--builtin_ntp_servers=127.27.1.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 18:29:09.652661 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:09.642215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38917 (local address 127.27.1.193:53157)
0807 18:29:09.643212 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:09.643257 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:09.644816 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:09.645404 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:09.645412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:09.645500 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:09.646035 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:09.646051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:09.647014 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:09.647017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:09.648349 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:09.648361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:09.649246 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:09.649254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:09.649592 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:09.650344 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:29:09.650347 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:09.650383 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:29:09.650456 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:29:09.650870 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:29:09.650881 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:29:09.650889 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 18:29:09.651086 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 18:29:09.651503 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":316,"threads_started":1}
I20260807 18:29:09.654081 5416 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38917
I20260807 18:29:09.654381 5416 heartbeater.cc:464] Registering TS with master...
I20260807 18:29:09.655167 5416 heartbeater.cc:511] Master 127.27.1.254:38917 requested a full tablet report, sending...
I20260807 18:29:09.657084 5212 ts_manager.cc:194] Registered new tserver with Master: a7a8dc2f19704f2a92a1223a5dcaa246 (127.27.1.193:35335)
I20260807 18:29:09.658527 5212 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.193:53157
W20260807 18:29:09.993559 5420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:09.993916 5420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:09.994081 5420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:10.001483 5420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:10.001636 5420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:10.001821 5420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:29:10.013096 5420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:10.014499 5420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:10.016451 5420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:10.030129 5428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:10.030174 5426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:10.030200 5425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:10.031030 5420 server_base.cc:1034] running on GCE node
I20260807 18:29:10.031668 5420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:10.032651 5420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:10.033885 5420 hybrid_clock.cc:648] HybridClock initialized: now 1786127350033831 us; error 45 us; skew 500 ppm
I20260807 18:29:10.034257 5420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:29:10.036664 5420 webserver.cc:460] Webserver started at http://127.27.1.194:33237/ using document root <none> and password file <none>
I20260807 18:29:10.037398 5420 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:10.037515 5420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:10.037868 5420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:10.040879 5420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "2aa366a2ff104233be00ffd938ac234a"
format_stamp: "Formatted at 2026-08-07 18:29:10 on dist-test-slave-1xrh"
server_key: "c1671ec7c2d1be538d09847d50c911d1"
server_key_iv: "c4291a7e7692b6a30a53f9f5575ea99c"
server_key_version: "encryptionkey@0"
I20260807 18:29:10.041674 5420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "2aa366a2ff104233be00ffd938ac234a"
format_stamp: "Formatted at 2026-08-07 18:29:10 on dist-test-slave-1xrh"
server_key: "c1671ec7c2d1be538d09847d50c911d1"
server_key_iv: "c4291a7e7692b6a30a53f9f5575ea99c"
server_key_version: "encryptionkey@0"
I20260807 18:29:10.046868 5420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 18:29:10.050354 5434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:10.051757 5420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:29:10.051947 5420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "2aa366a2ff104233be00ffd938ac234a"
format_stamp: "Formatted at 2026-08-07 18:29:10 on dist-test-slave-1xrh"
server_key: "c1671ec7c2d1be538d09847d50c911d1"
server_key_iv: "c4291a7e7692b6a30a53f9f5575ea99c"
server_key_version: "encryptionkey@0"
I20260807 18:29:10.052181 5420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:10.062906 5420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:10.063721 5420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:10.064036 5420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:10.064936 5420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:29:10.066248 5420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:29:10.066354 5420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:10.066478 5420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:29:10.066582 5420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:10.100857 5420 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:34441
I20260807 18:29:10.100939 5553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:34441 every 8 connection(s)
I20260807 18:29:10.102141 5420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:29:10.110785 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 5420
I20260807 18:29:10.110962 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:29:10.111588 27655 external_mini_cluster.cc:1428] Setting key eb4d34ede8fb9479a723ae577ae33bfb
I20260807 18:29:10.114816 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38917
--builtin_ntp_servers=127.27.1.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 18:29:10.115064 5418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:10.104674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:32841 (local address 127.27.1.254:38917)
0807 18:29:10.104856 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:10.104860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:10.106188 (+ 1328us) server_negotiation.cc:407] Connection header received
0807 18:29:10.107984 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:10.107990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:10.108083 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:10.108197 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:10.109206 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.109920 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.112227 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.112745 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.112871 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:10.114284 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:10.114288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:10.114291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:10.114307 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:10.114336 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:10.114348 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:10.114411 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:10.114703 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 18:29:10.114771 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:29:10.116518 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:10.105168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38917 (local address 127.27.1.194:32841)
0807 18:29:10.106031 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:10.106066 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:10.107723 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:10.108418 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:10.108427 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:10.108511 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:10.109037 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:10.109053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.110087 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.110093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:10.112043 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:10.112058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.112953 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.112958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:10.113161 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:10.113921 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:29:10.113925 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:10.113988 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:29:10.114090 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:29:10.114447 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:29:10.114462 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:29:10.114470 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 18:29:10.114694 (+ 224us) client_negotiation.cc:240] Negotiation successful
0807 18:29:10.115098 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":273,"threads_started":1}
I20260807 18:29:10.117861 5554 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38917
I20260807 18:29:10.118181 5554 heartbeater.cc:464] Registering TS with master...
I20260807 18:29:10.119060 5554 heartbeater.cc:511] Master 127.27.1.254:38917 requested a full tablet report, sending...
I20260807 18:29:10.120473 5212 ts_manager.cc:194] Registered new tserver with Master: 2aa366a2ff104233be00ffd938ac234a (127.27.1.194:34441)
I20260807 18:29:10.121254 5212 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.194:32841
W20260807 18:29:10.444109 5558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:10.444378 5558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:10.444519 5558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:10.451718 5558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:10.451812 5558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:10.451920 5558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:29:10.460973 5558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:10.462193 5558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:10.463979 5558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:10.476713 5566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:10.476779 5563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:10.476768 5564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:10.477535 5558 server_base.cc:1034] running on GCE node
I20260807 18:29:10.478113 5558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:10.479053 5558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:10.480302 5558 hybrid_clock.cc:648] HybridClock initialized: now 1786127350480246 us; error 50 us; skew 500 ppm
I20260807 18:29:10.480612 5558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:29:10.491452 5558 webserver.cc:460] Webserver started at http://127.27.1.195:45351/ using document root <none> and password file <none>
I20260807 18:29:10.492159 5558 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:10.492300 5558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:10.492637 5558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:10.495482 5558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "21b007df176b433795da997882568198"
format_stamp: "Formatted at 2026-08-07 18:29:10 on dist-test-slave-1xrh"
server_key: "e8d95ccbe9e6a1f3c4ce1ed7a4bd62d7"
server_key_iv: "ee77989283ff261b9e3a97ca282c1c87"
server_key_version: "encryptionkey@0"
I20260807 18:29:10.496273 5558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "21b007df176b433795da997882568198"
format_stamp: "Formatted at 2026-08-07 18:29:10 on dist-test-slave-1xrh"
server_key: "e8d95ccbe9e6a1f3c4ce1ed7a4bd62d7"
server_key_iv: "ee77989283ff261b9e3a97ca282c1c87"
server_key_version: "encryptionkey@0"
I20260807 18:29:10.501364 5558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:29:10.504637 5572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:10.505820 5558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 18:29:10.506011 5558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "21b007df176b433795da997882568198"
format_stamp: "Formatted at 2026-08-07 18:29:10 on dist-test-slave-1xrh"
server_key: "e8d95ccbe9e6a1f3c4ce1ed7a4bd62d7"
server_key_iv: "ee77989283ff261b9e3a97ca282c1c87"
server_key_version: "encryptionkey@0"
I20260807 18:29:10.506235 5558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:10.535236 5558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:10.536144 5558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:10.536468 5558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:10.537451 5558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:29:10.538918 5558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:29:10.539048 5558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:10.539202 5558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:29:10.539299 5558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:10.573313 5558 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:42249
I20260807 18:29:10.573402 5699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:42249 every 8 connection(s)
I20260807 18:29:10.574596 5558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:29:10.582427 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 5558
I20260807 18:29:10.582614 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:29:10.583240 27655 external_mini_cluster.cc:1428] Setting key c2f376e1c3cc8bd9eee434fd8e9748fd
I20260807 18:29:10.587469 5418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:10.577169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:58745 (local address 127.27.1.254:38917)
0807 18:29:10.577350 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:10.577355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:10.578699 (+ 1344us) server_negotiation.cc:407] Connection header received
0807 18:29:10.580491 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:10.580498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:10.580593 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:10.580704 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:10.581705 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.582457 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.584719 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.585245 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.585399 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:10.586771 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:10.586776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:10.586779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:10.586796 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:10.586827 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:10.586839 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:10.586908 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:10.587119 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 18:29:10.587241 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 18:29:10.588449 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:10.577682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38917 (local address 127.27.1.195:58745)
0807 18:29:10.578555 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:10.578589 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:10.580236 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:10.580918 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:10.580926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:10.581010 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:10.581536 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:10.581552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.582656 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.582662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:10.584579 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:10.584595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.585475 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.585481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:10.585681 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:10.586467 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:29:10.586471 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:10.586526 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:29:10.586598 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:29:10.586927 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:29:10.586935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:29:10.586941 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 18:29:10.587096 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 18:29:10.587428 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":285,"threads_started":1}
I20260807 18:29:10.589669 5701 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38917
I20260807 18:29:10.589943 5701 heartbeater.cc:464] Registering TS with master...
I20260807 18:29:10.590617 5701 heartbeater.cc:511] Master 127.27.1.254:38917 requested a full tablet report, sending...
I20260807 18:29:10.591915 5212 ts_manager.cc:194] Registered new tserver with Master: 21b007df176b433795da997882568198 (127.27.1.195:42249)
I20260807 18:29:10.592620 5212 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.195:58745
I20260807 18:29:10.598268 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 18:29:10.604349 5418 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:10.602702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.27.1.254:38917)
0807 18:29:10.602887 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:10.602892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:10.603394 (+ 502us) server_negotiation.cc:407] Connection header received
0807 18:29:10.603526 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:10.603529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:10.603599 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:10.603678 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:10.604090 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37996: BlockingRecv error: recv got EOF from 127.0.0.1:37996 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 18:29:10.605214 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 5280
I20260807 18:29:10.612561 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:10.606096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.27.1.193:35335)
0807 18:29:10.606607 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:10.606622 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:10.606648 (+ 26us) server_negotiation.cc:407] Connection header received
0807 18:29:10.606709 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:10.606715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:10.606890 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:10.607017 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:10.607394 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.608357 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.609631 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.610197 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.610296 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:10.611307 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:10.611315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:10.611326 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:10.611868 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:10.611916 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:10.611936 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:10.612033 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:10.612169 (+ 136us) server_negotiation.cc:299] Negotiation successful
0807 18:29:10.612286 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":282,"threads_started":1}
I20260807 18:29:10.613680 5410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:10.694818 5416 heartbeater.cc:503] Master 127.27.1.254:38917 was elected leader, sending a full tablet report...
I20260807 18:29:10.696719 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 5280
I20260807 18:29:10.715539 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 5420
I20260807 18:29:10.723450 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:10.716423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.27.1.194:34441)
0807 18:29:10.716982 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:10.716991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:10.717024 (+ 33us) server_negotiation.cc:407] Connection header received
0807 18:29:10.717121 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:10.717130 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:10.717378 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:10.717512 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:10.717883 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.718861 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.720144 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.720735 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.720846 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:10.721807 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:10.721818 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:10.721838 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:10.722557 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:10.722606 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:10.722627 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:10.722721 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:10.722870 (+ 149us) server_negotiation.cc:299] Negotiation successful
0807 18:29:10.723022 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":281,"threads_started":1}
I20260807 18:29:10.724757 5548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:10.807163 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 5420
I20260807 18:29:10.828166 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 5558
I20260807 18:29:10.836074 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:10.829054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.27.1.195:42249)
0807 18:29:10.829708 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:10.829715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:10.829749 (+ 34us) server_negotiation.cc:407] Connection header received
0807 18:29:10.829814 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:10.829820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:10.830052 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:10.830198 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:10.830573 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.831678 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.832842 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:10.833382 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:10.833480 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:10.834522 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:10.834529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:10.834543 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:10.835235 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:10.835284 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:10.835310 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:10.835400 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:10.835550 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 18:29:10.835709 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":316,"threads_started":1}
I20260807 18:29:10.837380 5695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:10.919305 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 5558
I20260807 18:29:10.939224 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 5179
I20260807 18:29:10.940492 5247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:11.047179 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 5179
2026-08-07T18:29:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2493 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T18:29:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:29:11Z Disabled control of system clock
I20260807 18:29:11.130411 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44071
--webserver_interface=127.27.1.254
--webserver_port=0
--builtin_ntp_servers=127.27.1.212:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:44071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 18:29:11.488196 5720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:11.488507 5720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:11.488683 5720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:11.496820 5720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:29:11.496919 5720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:11.496963 5720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:11.497012 5720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:29:11.497094 5720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:29:11.506168 5720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:44071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:11.507464 5720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:11.509238 5720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:11.521311 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:11.521274 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:11.521349 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:11.521962 5720 server_base.cc:1034] running on GCE node
I20260807 18:29:11.522778 5720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:11.524236 5720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:11.525441 5720 hybrid_clock.cc:648] HybridClock initialized: now 1786127351525405 us; error 28 us; skew 500 ppm
I20260807 18:29:11.525799 5720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:29:11.528223 5720 webserver.cc:460] Webserver started at http://127.27.1.254:41695/ using document root <none> and password file <none>
I20260807 18:29:11.528919 5720 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:11.529009 5720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:11.529381 5720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:11.534439 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "9359f97c63e849d9b44836e6c65084b6"
format_stamp: "Formatted at 2026-08-07 18:29:11 on dist-test-slave-1xrh"
server_key: "1e4289cda12c1300159fdee845ea9990"
server_key_iv: "b2211f87109df0bbe498b73a90280912"
server_key_version: "encryptionkey@0"
I20260807 18:29:11.535475 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "9359f97c63e849d9b44836e6c65084b6"
format_stamp: "Formatted at 2026-08-07 18:29:11 on dist-test-slave-1xrh"
server_key: "1e4289cda12c1300159fdee845ea9990"
server_key_iv: "b2211f87109df0bbe498b73a90280912"
server_key_version: "encryptionkey@0"
I20260807 18:29:11.541014 5720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260807 18:29:11.544755 5734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:11.546090 5720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 18:29:11.546296 5720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "9359f97c63e849d9b44836e6c65084b6"
format_stamp: "Formatted at 2026-08-07 18:29:11 on dist-test-slave-1xrh"
server_key: "1e4289cda12c1300159fdee845ea9990"
server_key_iv: "b2211f87109df0bbe498b73a90280912"
server_key_version: "encryptionkey@0"
I20260807 18:29:11.546525 5720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:11.552824 5720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:11.553649 5720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:11.553974 5720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:11.574862 5720 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:44071
I20260807 18:29:11.574882 5788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:44071 every 8 connection(s)
I20260807 18:29:11.576546 5720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:29:11.580214 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:29:11.583653 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 5720
I20260807 18:29:11.583894 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:29:11.584607 27655 external_mini_cluster.cc:1428] Setting key 3468a3e78b06392a3fb5f4c26fc0b3ba
I20260807 18:29:11.590920 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6: Bootstrap starting.
I20260807 18:29:11.593250 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:11.586631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.27.1.254:44071)
0807 18:29:11.587704 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:11.587715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:11.587803 (+ 88us) server_negotiation.cc:407] Connection header received
0807 18:29:11.589636 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:11.589659 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:11.590056 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:11.590378 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:11.591693 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:11.591699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:11.591711 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:11.591727 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:11.591785 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:11.591800 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:11.591982 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:11.592104 (+ 122us) server_negotiation.cc:299] Negotiation successful
0807 18:29:11.592388 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":392,"threads_started":1}
I20260807 18:29:11.594229 5789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6: Neither blocks nor log segments found. Creating new log.
I20260807 18:29:11.595592 5789 log.cc:824] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6: Log is configured to *not* fsync() on all Append() calls
I20260807 18:29:11.598165 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6: No bootstrap required, opened a new log
I20260807 18:29:11.605199 5789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9359f97c63e849d9b44836e6c65084b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44071 } }
I20260807 18:29:11.605541 5789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:29:11.605613 5789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9359f97c63e849d9b44836e6c65084b6, State: Initialized, Role: FOLLOWER
I20260807 18:29:11.606061 5789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9359f97c63e849d9b44836e6c65084b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44071 } }
I20260807 18:29:11.606245 5789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:29:11.606338 5789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:29:11.606458 5789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:29:11.608515 5789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9359f97c63e849d9b44836e6c65084b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44071 } }
I20260807 18:29:11.608930 5789 leader_election.cc:302] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9359f97c63e849d9b44836e6c65084b6; no voters:
I20260807 18:29:11.609444 5789 leader_election.cc:288] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:29:11.609597 5794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:29:11.610330 5794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [term 1 LEADER]: Becoming Leader. State: Replica: 9359f97c63e849d9b44836e6c65084b6, State: Running, Role: LEADER
I20260807 18:29:11.610880 5794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9359f97c63e849d9b44836e6c65084b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44071 } }
I20260807 18:29:11.611532 5789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:29:11.613932 5795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9359f97c63e849d9b44836e6c65084b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9359f97c63e849d9b44836e6c65084b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44071 } } }
I20260807 18:29:11.614315 5795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [sys.catalog]: This master's current role is: LEADER
I20260807 18:29:11.614025 5796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9359f97c63e849d9b44836e6c65084b6. Latest consensus state: current_term: 1 leader_uuid: "9359f97c63e849d9b44836e6c65084b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9359f97c63e849d9b44836e6c65084b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44071 } } }
I20260807 18:29:11.614440 5796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6 [sys.catalog]: This master's current role is: LEADER
I20260807 18:29:11.616174 5800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:29:11.621258 5800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:29:11.629546 5800 catalog_manager.cc:1347] Generated new cluster ID: 4c435f9ab4bf412c8da3203f01dd9949
I20260807 18:29:11.629647 5800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:29:11.669449 5800 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:29:11.669888 5800 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:29:11.677538 5800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9359f97c63e849d9b44836e6c65084b6: Generated new TSK 0
I20260807 18:29:11.678308 5800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:29:11.690470 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44071
--builtin_ntp_servers=127.27.1.212:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 18:29:12.046689 5815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:12.046969 5815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:12.047084 5815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:12.054382 5815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:12.054509 5815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:12.054690 5815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:29:12.063838 5815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:12.065105 5815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:12.066918 5815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:12.080205 5820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:12.080176 5823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:12.080204 5821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:12.081214 5815 server_base.cc:1034] running on GCE node
I20260807 18:29:12.081763 5815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:12.082599 5815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:12.083807 5815 hybrid_clock.cc:648] HybridClock initialized: now 1786127352083763 us; error 25 us; skew 500 ppm
I20260807 18:29:12.084122 5815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:29:12.086347 5815 webserver.cc:460] Webserver started at http://127.27.1.193:39341/ using document root <none> and password file <none>
I20260807 18:29:12.087030 5815 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:12.087180 5815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:12.087517 5815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:12.090359 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "eb774da61fef48c7954b30fe8938933a"
format_stamp: "Formatted at 2026-08-07 18:29:12 on dist-test-slave-1xrh"
server_key: "068d15adc60f59f6f2dc06505d5c7e36"
server_key_iv: "9d3c28a87508f5d5d2b56cbc47dbffa4"
server_key_version: "encryptionkey@0"
I20260807 18:29:12.091327 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "eb774da61fef48c7954b30fe8938933a"
format_stamp: "Formatted at 2026-08-07 18:29:12 on dist-test-slave-1xrh"
server_key: "068d15adc60f59f6f2dc06505d5c7e36"
server_key_iv: "9d3c28a87508f5d5d2b56cbc47dbffa4"
server_key_version: "encryptionkey@0"
I20260807 18:29:12.096549 5815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:29:12.099792 5830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:12.101014 5815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:29:12.101181 5815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "eb774da61fef48c7954b30fe8938933a"
format_stamp: "Formatted at 2026-08-07 18:29:12 on dist-test-slave-1xrh"
server_key: "068d15adc60f59f6f2dc06505d5c7e36"
server_key_iv: "9d3c28a87508f5d5d2b56cbc47dbffa4"
server_key_version: "encryptionkey@0"
I20260807 18:29:12.101389 5815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:12.106725 5815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:12.107591 5815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:12.107931 5815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:12.108827 5815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:29:12.110095 5815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:29:12.110181 5815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:12.110260 5815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:29:12.110319 5815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:12.143894 5815 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:40419
I20260807 18:29:12.144007 5952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:40419 every 8 connection(s)
I20260807 18:29:12.145226 5815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:29:12.152345 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 5815
I20260807 18:29:12.152572 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:29:12.153398 27655 external_mini_cluster.cc:1428] Setting key 2ca73f87ec2573dcd8f62c7a7776541c
I20260807 18:29:12.153589 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:12.147911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:58919 (local address 127.27.1.254:44071)
0807 18:29:12.148447 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:12.148452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:12.149478 (+ 1026us) server_negotiation.cc:407] Connection header received
0807 18:29:12.151202 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:12.151208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:12.151281 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:12.151406 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:12.152842 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:12.152847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:12.152850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:12.152865 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:12.152908 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:12.152919 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:12.153031 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:12.153266 (+ 235us) server_negotiation.cc:299] Negotiation successful
0807 18:29:12.153320 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":258,"threads_started":1}
I20260807 18:29:12.154567 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:12.148368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44071 (local address 127.27.1.193:58919)
0807 18:29:12.149311 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:12.149351 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:12.150910 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:12.151628 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:12.151636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:12.151720 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:12.152475 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:29:12.152480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:12.152536 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:29:12.152661 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:29:12.153026 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:29:12.153037 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:29:12.153045 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 18:29:12.153333 (+ 288us) client_negotiation.cc:240] Negotiation successful
0807 18:29:12.153592 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":280,"threads_started":1}
I20260807 18:29:12.155984 5953 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44071
I20260807 18:29:12.156277 5953 heartbeater.cc:464] Registering TS with master...
I20260807 18:29:12.156502 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44071
--builtin_ntp_servers=127.27.1.212:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 18:29:12.156867 5953 heartbeater.cc:511] Master 127.27.1.254:44071 requested a full tablet report, sending...
I20260807 18:29:12.158789 5752 ts_manager.cc:194] Registered new tserver with Master: eb774da61fef48c7954b30fe8938933a (127.27.1.193:40419)
W20260807 18:29:12.490688 5957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:12.491041 5957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:12.491217 5957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:12.498226 5957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:12.498338 5957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:12.498514 5957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:29:12.507388 5957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:12.508639 5957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:12.510429 5957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:12.523634 5962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:12.523761 5963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:12.523579 5966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:12.525187 5957 server_base.cc:1034] running on GCE node
I20260807 18:29:12.525758 5957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:12.526656 5957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:12.527874 5957 hybrid_clock.cc:648] HybridClock initialized: now 1786127352527833 us; error 48 us; skew 500 ppm
I20260807 18:29:12.528184 5957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:29:12.530445 5957 webserver.cc:460] Webserver started at http://127.27.1.194:41781/ using document root <none> and password file <none>
I20260807 18:29:12.531196 5957 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:12.531327 5957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:12.531652 5957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:12.534510 5957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "128a80b720e047ea87867d4085b51362"
format_stamp: "Formatted at 2026-08-07 18:29:12 on dist-test-slave-1xrh"
server_key: "1a992adfe44447a4db27cc9854124cb0"
server_key_iv: "432dbe752536019fd9de662fa3246e06"
server_key_version: "encryptionkey@0"
I20260807 18:29:12.535352 5957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "128a80b720e047ea87867d4085b51362"
format_stamp: "Formatted at 2026-08-07 18:29:12 on dist-test-slave-1xrh"
server_key: "1a992adfe44447a4db27cc9854124cb0"
server_key_iv: "432dbe752536019fd9de662fa3246e06"
server_key_version: "encryptionkey@0"
I20260807 18:29:12.540388 5957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 18:29:12.543685 5974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:12.544886 5957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:29:12.545064 5957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "128a80b720e047ea87867d4085b51362"
format_stamp: "Formatted at 2026-08-07 18:29:12 on dist-test-slave-1xrh"
server_key: "1a992adfe44447a4db27cc9854124cb0"
server_key_iv: "432dbe752536019fd9de662fa3246e06"
server_key_version: "encryptionkey@0"
I20260807 18:29:12.545289 5957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:12.550918 5957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:12.551848 5957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:12.552127 5957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:12.553005 5957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:29:12.554440 5957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:29:12.554563 5957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:12.554682 5957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:29:12.554766 5957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:12.588598 5957 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:45367
I20260807 18:29:12.588697 6094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:45367 every 8 connection(s)
I20260807 18:29:12.589906 5957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:29:12.594530 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 5957
I20260807 18:29:12.594734 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:29:12.595331 27655 external_mini_cluster.cc:1428] Setting key 30b300f5ce6e6d8ef10de6b27e38669a
I20260807 18:29:12.598430 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44071
--builtin_ntp_servers=127.27.1.212:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 18:29:12.598775 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:12.592621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:56423 (local address 127.27.1.254:44071)
0807 18:29:12.592822 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:12.592828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:12.594343 (+ 1515us) server_negotiation.cc:407] Connection header received
0807 18:29:12.596510 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:12.596515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:12.596598 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:12.596710 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:12.598056 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:12.598062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:12.598067 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:12.598085 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:12.598121 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:12.598132 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:12.598250 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:12.598445 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 18:29:12.598515 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 18:29:12.599757 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:12.593140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44071 (local address 127.27.1.194:56423)
0807 18:29:12.594140 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:12.594195 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:12.596261 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:12.596963 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:12.596971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:12.597056 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:12.597760 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:29:12.597762 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:12.597805 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:29:12.597888 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:29:12.598227 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:29:12.598238 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:29:12.598245 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 18:29:12.598520 (+ 275us) client_negotiation.cc:240] Negotiation successful
0807 18:29:12.598757 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":292,"threads_started":1}
I20260807 18:29:12.601166 6095 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44071
I20260807 18:29:12.601455 6095 heartbeater.cc:464] Registering TS with master...
I20260807 18:29:12.601984 6095 heartbeater.cc:511] Master 127.27.1.254:44071 requested a full tablet report, sending...
I20260807 18:29:12.603406 5752 ts_manager.cc:194] Registered new tserver with Master: 128a80b720e047ea87867d4085b51362 (127.27.1.194:45367)
W20260807 18:29:12.938153 6099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:12.938426 6099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:12.938515 6099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:12.945663 6099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:12.945760 6099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:12.945881 6099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:29:12.955425 6099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:12.956684 6099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:12.958475 6099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:12.971637 6105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:12.971594 6109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:12.971688 6106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:12.976495 6099 server_base.cc:1034] running on GCE node
I20260807 18:29:12.977034 6099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:12.977907 6099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:12.979182 6099 hybrid_clock.cc:648] HybridClock initialized: now 1786127352979080 us; error 36 us; skew 500 ppm
I20260807 18:29:12.979534 6099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:29:12.981931 6099 webserver.cc:460] Webserver started at http://127.27.1.195:34811/ using document root <none> and password file <none>
I20260807 18:29:12.982632 6099 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:12.982753 6099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:12.983098 6099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:12.986327 6099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "b90d732c731f46318bca3a096fc8a84d"
format_stamp: "Formatted at 2026-08-07 18:29:12 on dist-test-slave-1xrh"
server_key: "85511ab06f388bd0ede040a67da6401d"
server_key_iv: "b00a472e917acc96107b6b352a4fa233"
server_key_version: "encryptionkey@0"
I20260807 18:29:12.987267 6099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "b90d732c731f46318bca3a096fc8a84d"
format_stamp: "Formatted at 2026-08-07 18:29:12 on dist-test-slave-1xrh"
server_key: "85511ab06f388bd0ede040a67da6401d"
server_key_iv: "b00a472e917acc96107b6b352a4fa233"
server_key_version: "encryptionkey@0"
I20260807 18:29:12.992427 6099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 18:29:12.995880 6115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:12.997196 6099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:29:12.997357 6099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "b90d732c731f46318bca3a096fc8a84d"
format_stamp: "Formatted at 2026-08-07 18:29:12 on dist-test-slave-1xrh"
server_key: "85511ab06f388bd0ede040a67da6401d"
server_key_iv: "b00a472e917acc96107b6b352a4fa233"
server_key_version: "encryptionkey@0"
I20260807 18:29:12.997572 6099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:13.003356 6099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:13.004145 6099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:13.004441 6099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:13.005383 6099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:29:13.006767 6099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:29:13.006884 6099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:13.007014 6099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:29:13.007102 6099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:13.041460 6099 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:43695
I20260807 18:29:13.041551 6237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:43695 every 8 connection(s)
I20260807 18:29:13.042814 6099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:29:13.046567 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 6099
I20260807 18:29:13.046759 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:29:13.047312 27655 external_mini_cluster.cc:1428] Setting key af7b309a4512a1fac7ca6a8c578c6a37
I20260807 18:29:13.051599 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:13.045377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:39581 (local address 127.27.1.254:44071)
0807 18:29:13.045564 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:13.045568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:13.047020 (+ 1452us) server_negotiation.cc:407] Connection header received
0807 18:29:13.049035 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:13.049041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:13.049122 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:13.049234 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:13.050781 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:13.050787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:13.050792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:13.050810 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:13.050852 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:13.050864 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:13.050990 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:13.051215 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 18:29:13.051287 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:29:13.052474 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:13.045878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44071 (local address 127.27.1.195:39581)
0807 18:29:13.046871 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:13.046915 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:13.048754 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:13.049474 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:13.049483 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:13.049570 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:13.050435 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:29:13.050440 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:13.050516 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:29:13.050626 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:29:13.050959 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:29:13.050970 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:29:13.050978 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 18:29:13.051297 (+ 319us) client_negotiation.cc:240] Negotiation successful
0807 18:29:13.051551 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":293,"threads_started":1}
I20260807 18:29:13.053686 6238 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44071
I20260807 18:29:13.053984 6238 heartbeater.cc:464] Registering TS with master...
I20260807 18:29:13.054572 6238 heartbeater.cc:511] Master 127.27.1.254:44071 requested a full tablet report, sending...
I20260807 18:29:13.055962 5752 ts_manager.cc:194] Registered new tserver with Master: b90d732c731f46318bca3a096fc8a84d (127.27.1.195:43695)
I20260807 18:29:13.062829 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 18:29:13.069656 5955 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:29:13.068043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.27.1.254:44071)
0807 18:29:13.068206 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:13.068211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:13.068714 (+ 503us) server_negotiation.cc:407] Connection header received
0807 18:29:13.068849 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:13.068855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:13.068913 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:13.068998 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:13.069339 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60378: BlockingRecv error: recv got EOF from 127.0.0.1:60378 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:29:13.070590 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 5815
I20260807 18:29:13.074151 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:13.071608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.27.1.193:40419)
0807 18:29:13.072287 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:13.072296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:13.072325 (+ 29us) server_negotiation.cc:407] Connection header received
0807 18:29:13.072395 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:13.072401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:13.072596 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:13.072725 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:13.073539 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:13.073546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:13.073557 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:13.073575 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:13.073620 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:13.073634 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:13.073745 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:13.073824 (+ 79us) server_negotiation.cc:299] Negotiation successful
0807 18:29:13.073905 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":381,"threads_started":1}
I20260807 18:29:13.075371 5947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:13.164844 5953 heartbeater.cc:503] Master 127.27.1.254:44071 was elected leader, sending a full tablet report...
I20260807 18:29:13.166836 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 5815
I20260807 18:29:13.188194 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 5957
I20260807 18:29:13.192404 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:13.189203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.27.1.194:45367)
0807 18:29:13.189855 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:13.189867 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:13.189904 (+ 37us) server_negotiation.cc:407] Connection header received
0807 18:29:13.189995 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:13.190007 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:13.190303 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:13.190474 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:13.191574 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:13.191583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:13.191600 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:13.191626 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:13.191683 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:13.191703 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:13.191845 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:13.191913 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 18:29:13.192025 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":310,"threads_started":1}
I20260807 18:29:13.193785 6090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:13.281312 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 5957
I20260807 18:29:13.300727 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 6099
I20260807 18:29:13.304750 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:13.301758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.27.1.195:43695)
0807 18:29:13.302556 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:13.302562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:13.302593 (+ 31us) server_negotiation.cc:407] Connection header received
0807 18:29:13.302658 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:13.302665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:13.302887 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:13.303014 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:13.304020 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:13.304026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:13.304037 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:29:13.304055 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:13.304092 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:29:13.304106 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:13.304243 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:13.304304 (+ 61us) server_negotiation.cc:299] Negotiation successful
0807 18:29:13.304425 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":407,"threads_started":1}
I20260807 18:29:13.306037 6233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:13.390902 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 6099
I20260807 18:29:13.411077 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 5720
I20260807 18:29:13.412380 5784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:13.512987 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 5720
2026-08-07T18:29:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2465 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:29:13 dist-test-slave-1xrh krb5kdc[6261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:29:13 dist-test-slave-1xrh krb5kdc[6261](info): set up 2 sockets
Aug 07 18:29:13 dist-test-slave-1xrh krb5kdc[6261](info): commencing operation
krb5kdc: starting...
W20260807 18:29:14.674852 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:29:14 dist-test-slave-1xrh krb5kdc[6261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:29:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:29:14.950242 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:42909
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:42909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:29:15.285193 6275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:15.285504 6275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:15.285691 6275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:15.292730 6275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:29:15.292867 6275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:15.292965 6275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:15.293074 6275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:29:15.293157 6275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:29:15.302630 6275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:42909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:42909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:15.303997 6275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:15.305270 6275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:15.318717 6281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:15.318722 6280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:15.318899 6283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:15.321018 6275 server_base.cc:1034] running on GCE node
I20260807 18:29:15.321801 6275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:15.323241 6275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:15.324437 6275 hybrid_clock.cc:648] HybridClock initialized: now 1786127355324391 us; error 33 us; skew 500 ppm
I20260807 18:29:15.324792 6275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:29:15 dist-test-slave-1xrh krb5kdc[6261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127355, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:29:15.328404 6275 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:29:15.329907 6275 webserver.cc:460] Webserver started at http://127.27.1.254:36227/ using document root <none> and password file <none>
I20260807 18:29:15.330616 6275 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:15.330758 6275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:15.331058 6275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:15.334054 6275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "b2cb27ea4e2942f28095ff9454c4c45f"
format_stamp: "Formatted at 2026-08-07 18:29:15 on dist-test-slave-1xrh"
server_key: "c331907e2085e8f666d4a30df1013707"
server_key_iv: "f4dccb264639c0f4b797c572ab6ae4b6"
server_key_version: "encryptionkey@0"
I20260807 18:29:15.334853 6275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "b2cb27ea4e2942f28095ff9454c4c45f"
format_stamp: "Formatted at 2026-08-07 18:29:15 on dist-test-slave-1xrh"
server_key: "c331907e2085e8f666d4a30df1013707"
server_key_iv: "f4dccb264639c0f4b797c572ab6ae4b6"
server_key_version: "encryptionkey@0"
I20260807 18:29:15.340392 6275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 18:29:15.343833 6291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:15.345168 6275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:29:15.345366 6275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "b2cb27ea4e2942f28095ff9454c4c45f"
format_stamp: "Formatted at 2026-08-07 18:29:15 on dist-test-slave-1xrh"
server_key: "c331907e2085e8f666d4a30df1013707"
server_key_iv: "f4dccb264639c0f4b797c572ab6ae4b6"
server_key_version: "encryptionkey@0"
I20260807 18:29:15.345543 6275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:15.388507 6275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:15.392863 6275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:15.393117 6275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:15.411837 6275 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:42909
I20260807 18:29:15.411878 6349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:42909 every 8 connection(s)
I20260807 18:29:15.413200 6275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:29:15.416563 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 6275
I20260807 18:29:15.416803 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:29:15.416939 6350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:29:15.417514 27655 external_mini_cluster.cc:1428] Setting key e91bba540aafc2dc4cfe8927db2b1d2d
I20260807 18:29:15.431375 6350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f: Bootstrap starting.
Aug 07 18:29:15 dist-test-slave-1xrh krb5kdc[6261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:29:15.435070 6350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f: Neither blocks nor log segments found. Creating new log.
I20260807 18:29:15.436519 6350 log.cc:824] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f: Log is configured to *not* fsync() on all Append() calls
I20260807 18:29:15.440440 6350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f: No bootstrap required, opened a new log
I20260807 18:29:15.442569 6354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:15.419576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.27.1.254:42909)
0807 18:29:15.420391 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:15.420407 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:15.420460 (+ 53us) server_negotiation.cc:407] Connection header received
0807 18:29:15.422059 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:15.422093 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:15.422561 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:15.423003 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:15.424416 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:15.426344 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:15.427727 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:15.428847 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:15.429044 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:15.433071 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:15.433084 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:15.433103 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:15.433152 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:15.437407 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:15.438278 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:15.438288 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:15.438296 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:15.438525 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:15.438906 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:15.438911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:15.438916 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:15.439770 (+ 854us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:15.440118 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:15.440433 (+ 315us) server_negotiation.cc:299] Negotiation successful
0807 18:29:15.441032 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":350,"threads_started":1}
I20260807 18:29:15.450217 6350 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cb27ea4e2942f28095ff9454c4c45f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42909 } }
I20260807 18:29:15.450635 6350 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:29:15.450790 6350 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2cb27ea4e2942f28095ff9454c4c45f, State: Initialized, Role: FOLLOWER
I20260807 18:29:15.451457 6350 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cb27ea4e2942f28095ff9454c4c45f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42909 } }
I20260807 18:29:15.451725 6350 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:29:15.451879 6350 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:29:15.452065 6350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:29:15.454727 6350 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cb27ea4e2942f28095ff9454c4c45f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42909 } }
I20260807 18:29:15.455394 6350 leader_election.cc:302] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2cb27ea4e2942f28095ff9454c4c45f; no voters:
I20260807 18:29:15.456038 6350 leader_election.cc:288] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:29:15.456197 6356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:29:15.457005 6356 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [term 1 LEADER]: Becoming Leader. State: Replica: b2cb27ea4e2942f28095ff9454c4c45f, State: Running, Role: LEADER
I20260807 18:29:15.457600 6356 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cb27ea4e2942f28095ff9454c4c45f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42909 } }
I20260807 18:29:15.458515 6350 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:29:15.460845 6357 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2cb27ea4e2942f28095ff9454c4c45f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cb27ea4e2942f28095ff9454c4c45f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42909 } } }
I20260807 18:29:15.460778 6358 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2cb27ea4e2942f28095ff9454c4c45f. Latest consensus state: current_term: 1 leader_uuid: "b2cb27ea4e2942f28095ff9454c4c45f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cb27ea4e2942f28095ff9454c4c45f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42909 } } }
I20260807 18:29:15.461210 6358 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [sys.catalog]: This master's current role is: LEADER
I20260807 18:29:15.461210 6357 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f [sys.catalog]: This master's current role is: LEADER
I20260807 18:29:15.463529 6363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:29:15.469157 6363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:29:15.478816 6363 catalog_manager.cc:1347] Generated new cluster ID: d19d7d12cdc8432ba2446dca6297248e
I20260807 18:29:15.478935 6363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:29:15.494491 6363 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:29:15.495826 6363 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:29:15.502751 6363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f: Generated new TSK 0
I20260807 18:29:15.503599 6363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:29:15.594801 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42909
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:29:15.962136 6378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:15.962515 6378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:15.962617 6378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:15.970090 6378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:15.970204 6378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:15.970360 6378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:29:15.980197 6378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:15.981537 6378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:15.982867 6378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:15.996035 6386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:15.996114 6384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:15.996075 6383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:15.996997 6378 server_base.cc:1034] running on GCE node
I20260807 18:29:15.997610 6378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:15.998556 6378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:15.999775 6378 hybrid_clock.cc:648] HybridClock initialized: now 1786127355999728 us; error 57 us; skew 500 ppm
I20260807 18:29:16.000102 6378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:29:16 dist-test-slave-1xrh krb5kdc[6261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127356, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:29:16.008992 6378 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:29:16.010529 6378 webserver.cc:460] Webserver started at http://127.27.1.193:35777/ using document root <none> and password file <none>
I20260807 18:29:16.011312 6378 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:16.011456 6378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:16.011758 6378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:16.014663 6378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "1dc6db92d2af41ce941dd64a162e3a80"
format_stamp: "Formatted at 2026-08-07 18:29:16 on dist-test-slave-1xrh"
server_key: "50a3d9d354fe62a25b88b965e9a08985"
server_key_iv: "5e1dc773bebbe68fa2ec41d6ee9709c5"
server_key_version: "encryptionkey@0"
I20260807 18:29:16.015475 6378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "1dc6db92d2af41ce941dd64a162e3a80"
format_stamp: "Formatted at 2026-08-07 18:29:16 on dist-test-slave-1xrh"
server_key: "50a3d9d354fe62a25b88b965e9a08985"
server_key_iv: "5e1dc773bebbe68fa2ec41d6ee9709c5"
server_key_version: "encryptionkey@0"
I20260807 18:29:16.020630 6378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 18:29:16.023988 6393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:16.025213 6378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 18:29:16.025389 6378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "1dc6db92d2af41ce941dd64a162e3a80"
format_stamp: "Formatted at 2026-08-07 18:29:16 on dist-test-slave-1xrh"
server_key: "50a3d9d354fe62a25b88b965e9a08985"
server_key_iv: "5e1dc773bebbe68fa2ec41d6ee9709c5"
server_key_version: "encryptionkey@0"
I20260807 18:29:16.025599 6378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:16.043569 6378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:16.047429 6378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:16.047710 6378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:16.048681 6378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:29:16.050190 6378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:29:16.050290 6378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:16.050431 6378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:29:16.050501 6378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:16.083379 6378 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:34469
I20260807 18:29:16.083416 6522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:34469 every 8 connection(s)
I20260807 18:29:16.084645 6378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:29:16.089037 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 6378
I20260807 18:29:16.089284 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:29:16.089907 27655 external_mini_cluster.cc:1428] Setting key 7a89f3f97ed4488871a2934fc38aa3af
Aug 07 18:29:16 dist-test-slave-1xrh krb5kdc[6261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127356, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:29:16.104421 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:16.087234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:60537 (local address 127.27.1.254:42909)
0807 18:29:16.087691 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:16.087696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:16.088658 (+ 962us) server_negotiation.cc:407] Connection header received
0807 18:29:16.090610 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:16.090616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:16.090707 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:16.090823 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:16.092339 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:16.093041 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:16.094628 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:16.095021 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:16.095179 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:16.098759 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:16.098764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:16.098767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:16.098797 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:16.101399 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:16.102105 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:16.102115 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:16.102118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:16.102232 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:16.102659 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:16.102664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:16.102671 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:16.102941 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:16.103095 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:16.103826 (+ 731us) server_negotiation.cc:299] Negotiation successful
0807 18:29:16.104062 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20260807 18:29:16.105681 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:16.087782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42909 (local address 127.27.1.193:60537)
0807 18:29:16.088490 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:16.088533 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:16.090336 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:16.091055 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:16.091063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:16.091628 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:16.092176 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:16.092193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:16.093208 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:16.093212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:16.094468 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:16.094480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:16.095285 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:16.095293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:16.095522 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:16.096196 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:16.096222 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:16.098547 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:29:16.101538 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:29:16.101547 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:29:16.101563 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:29:16.101979 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:29:16.102341 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:29:16.102346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:29:16.102350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:29:16.102541 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:29:16.103068 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:29:16.103075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:29:16.103430 (+ 355us) client_negotiation.cc:769] Sending connection context
0807 18:29:16.103694 (+ 264us) client_negotiation.cc:240] Negotiation successful
0807 18:29:16.104225 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":286,"threads_started":1}
I20260807 18:29:16.107249 6523 heartbeater.cc:347] Connected to a master server at 127.27.1.254:42909
I20260807 18:29:16.107558 6523 heartbeater.cc:464] Registering TS with master...
I20260807 18:29:16.108294 6523 heartbeater.cc:511] Master 127.27.1.254:42909 requested a full tablet report, sending...
I20260807 18:29:16.110340 6310 ts_manager.cc:194] Registered new tserver with Master: 1dc6db92d2af41ce941dd64a162e3a80 (127.27.1.193:34469)
I20260807 18:29:16.111843 6310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:60537
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:29:16.151854 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42909
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:29:16.509974 6529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:16.510253 6529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:16.510442 6529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:16.517776 6529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:16.517874 6529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:16.518059 6529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:29:16.527539 6529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:16.528790 6529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:16.530128 6529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:16.542963 6534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:16.543007 6538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:16.543037 6535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:16.544230 6529 server_base.cc:1034] running on GCE node
I20260807 18:29:16.544895 6529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:16.545785 6529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:16.547025 6529 hybrid_clock.cc:648] HybridClock initialized: now 1786127356546983 us; error 46 us; skew 500 ppm
I20260807 18:29:16.547384 6529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:29:16 dist-test-slave-1xrh krb5kdc[6261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127356, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:29:16.550563 6529 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:29:16.552035 6529 webserver.cc:460] Webserver started at http://127.27.1.194:37803/ using document root <none> and password file <none>
I20260807 18:29:16.552719 6529 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:16.552860 6529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:16.553157 6529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:16.556003 6529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "ed5217e9952d48588bcd2c357d6fb4c0"
format_stamp: "Formatted at 2026-08-07 18:29:16 on dist-test-slave-1xrh"
server_key: "b76fc86bd3efeffded2e31dfdb951d72"
server_key_iv: "2f90eb308e6d0d9053b4dc1afa5bc01d"
server_key_version: "encryptionkey@0"
I20260807 18:29:16.556753 6529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "ed5217e9952d48588bcd2c357d6fb4c0"
format_stamp: "Formatted at 2026-08-07 18:29:16 on dist-test-slave-1xrh"
server_key: "b76fc86bd3efeffded2e31dfdb951d72"
server_key_iv: "2f90eb308e6d0d9053b4dc1afa5bc01d"
server_key_version: "encryptionkey@0"
I20260807 18:29:16.561908 6529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:29:16.565250 6545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:16.566505 6529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260807 18:29:16.566679 6529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "ed5217e9952d48588bcd2c357d6fb4c0"
format_stamp: "Formatted at 2026-08-07 18:29:16 on dist-test-slave-1xrh"
server_key: "b76fc86bd3efeffded2e31dfdb951d72"
server_key_iv: "2f90eb308e6d0d9053b4dc1afa5bc01d"
server_key_version: "encryptionkey@0"
I20260807 18:29:16.566869 6529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:16.581866 6529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:16.585817 6529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:16.586175 6529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:16.587088 6529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:29:16.588368 6529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:29:16.588438 6529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:16.588539 6529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:29:16.588595 6529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:16.623615 6529 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:44353
I20260807 18:29:16.623687 6667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:44353 every 8 connection(s)
I20260807 18:29:16.625200 6529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:29:16.629253 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 6529
I20260807 18:29:16.629462 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:29:16.630036 27655 external_mini_cluster.cc:1428] Setting key 9d45e241f9c5c5d7c7041bf5f1bf3758
Aug 07 18:29:16 dist-test-slave-1xrh krb5kdc[6261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127356, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:29:16.648615 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:16.627769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:59519 (local address 127.27.1.254:42909)
0807 18:29:16.628222 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:16.628226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:16.629359 (+ 1133us) server_negotiation.cc:407] Connection header received
0807 18:29:16.631661 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:16.631665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:16.631727 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:16.631821 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:16.633813 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:16.634565 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:16.636991 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:16.637410 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:16.637501 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:16.641947 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:16.641951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:16.641956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:16.641988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:16.645265 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:16.645954 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:16.645963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:16.645969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:16.646128 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:16.646540 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:16.646548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:16.646554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:16.646942 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:16.647104 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:16.647851 (+ 747us) server_negotiation.cc:299] Negotiation successful
0807 18:29:16.648114 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":246,"threads_started":1}
I20260807 18:29:16.649248 6671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:16.628245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42909 (local address 127.27.1.194:59519)
0807 18:29:16.629166 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:16.629228 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:16.631381 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:16.632095 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:16.632102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:16.632786 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:16.633621 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:16.633650 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:16.634725 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:16.634733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:16.636838 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:16.636855 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:16.637625 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:16.637634 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:16.637955 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:16.638690 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:16.638723 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:16.641704 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:29:16.645403 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:29:16.645410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:29:16.645421 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:29:16.645819 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:29:16.646250 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:29:16.646254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:29:16.646256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:29:16.646430 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:29:16.647081 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:29:16.647088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:29:16.647473 (+ 385us) client_negotiation.cc:769] Sending connection context
0807 18:29:16.647691 (+ 218us) client_negotiation.cc:240] Negotiation successful
0807 18:29:16.648067 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":281,"threads_started":1}
I20260807 18:29:16.650751 6668 heartbeater.cc:347] Connected to a master server at 127.27.1.254:42909
I20260807 18:29:16.651048 6668 heartbeater.cc:464] Registering TS with master...
I20260807 18:29:16.651788 6668 heartbeater.cc:511] Master 127.27.1.254:42909 requested a full tablet report, sending...
I20260807 18:29:16.653195 6310 ts_manager.cc:194] Registered new tserver with Master: ed5217e9952d48588bcd2c357d6fb4c0 (127.27.1.194:44353)
I20260807 18:29:16.654006 6310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:59519
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:29:16.693850 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42909
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:29:17.019531 6674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:29:17.019841 6674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:29:17.020017 6674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:29:17.027042 6674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:29:17.027211 6674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:29:17.027446 6674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:29:17.038712 6674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:42909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:29:17.040055 6674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:29:17.041348 6674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:29:17.054553 6680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:17.054488 6683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:29:17.054522 6681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:29:17.055696 6674 server_base.cc:1034] running on GCE node
I20260807 18:29:17.056295 6674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:29:17.057199 6674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:29:17.058432 6674 hybrid_clock.cc:648] HybridClock initialized: now 1786127357058360 us; error 54 us; skew 500 ppm
I20260807 18:29:17.058741 6674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:29:17 dist-test-slave-1xrh krb5kdc[6261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127357, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:29:17.061877 6674 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:29:17.063333 6674 webserver.cc:460] Webserver started at http://127.27.1.195:43407/ using document root <none> and password file <none>
I20260807 18:29:17.063994 6674 fs_manager.cc:359] Metadata directory not provided
I20260807 18:29:17.064092 6674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:29:17.064339 6674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:29:17.067240 6674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "1bc6dfd4725a46559264faf38005abf9"
format_stamp: "Formatted at 2026-08-07 18:29:17 on dist-test-slave-1xrh"
server_key: "caa172e2d96ae029b8bc9c2033a6b9a1"
server_key_iv: "03c8f85de0ae64b5e867215c0c5258e5"
server_key_version: "encryptionkey@0"
I20260807 18:29:17.068027 6674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "1bc6dfd4725a46559264faf38005abf9"
format_stamp: "Formatted at 2026-08-07 18:29:17 on dist-test-slave-1xrh"
server_key: "caa172e2d96ae029b8bc9c2033a6b9a1"
server_key_iv: "03c8f85de0ae64b5e867215c0c5258e5"
server_key_version: "encryptionkey@0"
I20260807 18:29:17.073278 6674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260807 18:29:17.076515 6691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:17.077855 6674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 18:29:17.077982 6674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "1bc6dfd4725a46559264faf38005abf9"
format_stamp: "Formatted at 2026-08-07 18:29:17 on dist-test-slave-1xrh"
server_key: "caa172e2d96ae029b8bc9c2033a6b9a1"
server_key_iv: "03c8f85de0ae64b5e867215c0c5258e5"
server_key_version: "encryptionkey@0"
I20260807 18:29:17.078138 6674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:29:17.094200 6674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:29:17.097924 6674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:29:17.098167 6674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:29:17.099021 6674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:29:17.100291 6674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:29:17.100364 6674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:17.100467 6674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:29:17.100512 6674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:29:17.114593 6523 heartbeater.cc:503] Master 127.27.1.254:42909 was elected leader, sending a full tablet report...
I20260807 18:29:17.134325 6674 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:32889
I20260807 18:29:17.134372 6813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:32889 every 8 connection(s)
I20260807 18:29:17.135627 6674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:29:17.139012 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 6674
I20260807 18:29:17.139261 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:29:17.139789 27655 external_mini_cluster.cc:1428] Setting key e08b58c8f340ca039296b60a198c938b
Aug 07 18:29:17 dist-test-slave-1xrh krb5kdc[6261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127357, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:29:17.155782 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.138047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:36611 (local address 127.27.1.254:42909)
0807 18:29:17.138211 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.138215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.139646 (+ 1431us) server_negotiation.cc:407] Connection header received
0807 18:29:17.141695 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.141701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.141804 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.141905 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:17.143685 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.144419 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.145880 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.146262 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.146354 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.149784 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:17.149789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:17.149795 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:17.149843 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:17.152642 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:17.153368 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:17.153372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:17.153375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:17.153497 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:17.153988 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:17.153992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:17.153995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:17.154331 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:17.154494 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.155371 (+ 877us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.155548 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 18:29:17.156710 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.138546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:42909 (local address 127.27.1.195:36611)
0807 18:29:17.139491 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:17.139533 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:17.141392 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:17.142187 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:17.142198 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:17.142915 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:29:17.143527 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.143544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.144560 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.144564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.145755 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.145767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.146443 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.146447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.146654 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.147329 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:29:17.147350 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:29:17.149558 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:29:17.152792 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:29:17.152799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:29:17.152810 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:29:17.153232 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:29:17.153607 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:29:17.153612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:29:17.153615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:29:17.153861 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:29:17.154475 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:29:17.154484 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:29:17.154903 (+ 419us) client_negotiation.cc:769] Sending connection context
0807 18:29:17.155259 (+ 356us) client_negotiation.cc:240] Negotiation successful
0807 18:29:17.155639 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":345,"threads_started":1}
I20260807 18:29:17.157913 6814 heartbeater.cc:347] Connected to a master server at 127.27.1.254:42909
I20260807 18:29:17.158232 6814 heartbeater.cc:464] Registering TS with master...
I20260807 18:29:17.159027 6814 heartbeater.cc:511] Master 127.27.1.254:42909 requested a full tablet report, sending...
I20260807 18:29:17.160476 6310 ts_manager.cc:194] Registered new tserver with Master: 1bc6dfd4725a46559264faf38005abf9 (127.27.1.195:32889)
I20260807 18:29:17.161274 6310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:36611
I20260807 18:29:17.167073 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:29:17.184144 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.171913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.27.1.254:42909)
0807 18:29:17.172077 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.172081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.172526 (+ 445us) server_negotiation.cc:407] Connection header received
0807 18:29:17.172659 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.172663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.172729 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.172807 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:17.174128 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.174834 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.176035 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.176427 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.176528 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.178615 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:17.178619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:17.178622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:17.178652 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:17.181787 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:17.182429 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:17.182437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:17.182442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:17.182585 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:17.182889 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:17.182892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:17.182894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:17.183254 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:17.183451 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.183709 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.183877 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 18:29:17.188102 6310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:29:17.195201 6310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:29:17.219445 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.211272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:32889 (local address 127.0.0.1:41450)
0807 18:29:17.212734 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:17.212747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:17.212873 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:17.213344 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:17.213350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:17.213372 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:29:17.213764 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.213779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.215436 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.215442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.217002 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.217018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.218658 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.218664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.218834 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.218863 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 18:29:17.218940 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 18:29:17.219068 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"mutex_wait_us":26,"thread_start_us":1236,"threads_started":1}
I20260807 18:29:17.219771 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.208211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.27.1.194:44353)
0807 18:29:17.209723 (+ 1512us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.209736 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.210335 (+ 599us) server_negotiation.cc:407] Connection header received
0807 18:29:17.210665 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.210673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.210945 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.211167 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:29:17.212380 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.213850 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.216984 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.218108 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.218254 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.218449 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.219251 (+ 802us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.219419 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":1101,"threads_started":1}
I20260807 18:29:17.220286 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.207657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:44353 (local address 127.0.0.1:38948)
0807 18:29:17.210231 (+ 2574us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:17.210252 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:17.210440 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:17.211460 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:17.211467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:17.211519 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:29:17.212164 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.212181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.214028 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.214035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.216811 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.216827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.218319 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.218326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.218546 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.218608 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 18:29:17.218752 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 18:29:17.218930 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2437,"mutex_wait_us":349,"thread_start_us":843,"threads_started":1}
I20260807 18:29:17.220497 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.210604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.27.1.195:32889)
0807 18:29:17.211975 (+ 1371us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.211983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.212839 (+ 856us) server_negotiation.cc:407] Connection header received
0807 18:29:17.213028 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.213033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.213173 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.213339 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:29:17.213964 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.215305 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.217158 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.218134 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.218284 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.218507 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.219688 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.220075 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":992,"threads_started":1}
I20260807 18:29:17.220736 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.206911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34469 (local address 127.0.0.1:44276)
0807 18:29:17.208713 (+ 1802us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:17.208763 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:17.208970 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:17.211229 (+ 2259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:17.211237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:17.211269 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:29:17.211897 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.211912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.213860 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.213865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.216811 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.216827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.219888 (+ 3061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.219894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.220085 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.220126 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 18:29:17.220210 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 18:29:17.220358 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1478,"mutex_wait_us":178,"thread_start_us":3014,"threads_started":1}
I20260807 18:29:17.221041 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.207322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.27.1.193:34469)
0807 18:29:17.210152 (+ 2830us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.210159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.210187 (+ 28us) server_negotiation.cc:407] Connection header received
0807 18:29:17.210287 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.210295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.210946 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.211213 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:29:17.212100 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.213693 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.218552 (+ 4859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.219698 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.219824 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.220033 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.220518 (+ 485us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.220685 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2658,"thread_start_us":2511,"threads_started":1}
I20260807 18:29:17.224082 6741 tablet_service.cc:1467] Processing CreateTablet for tablet a9a8da76a8754992a0e18b70e81ea178 (DEFAULT_TABLE table=test-table [id=e62227e4dc5e4b30b810e3b55a3bbd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:29:17.224136 6597 tablet_service.cc:1467] Processing CreateTablet for tablet a9a8da76a8754992a0e18b70e81ea178 (DEFAULT_TABLE table=test-table [id=e62227e4dc5e4b30b810e3b55a3bbd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:29:17.224476 6449 tablet_service.cc:1467] Processing CreateTablet for tablet a9a8da76a8754992a0e18b70e81ea178 (DEFAULT_TABLE table=test-table [id=e62227e4dc5e4b30b810e3b55a3bbd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:29:17.225338 6741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a8da76a8754992a0e18b70e81ea178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:29:17.225518 6597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a8da76a8754992a0e18b70e81ea178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:29:17.225706 6449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a8da76a8754992a0e18b70e81ea178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:29:17.237574 6835 tablet_bootstrap.cc:492] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: Bootstrap starting.
I20260807 18:29:17.240047 6836 tablet_bootstrap.cc:492] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: Bootstrap starting.
I20260807 18:29:17.241039 6837 tablet_bootstrap.cc:492] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: Bootstrap starting.
I20260807 18:29:17.241341 6835 tablet_bootstrap.cc:654] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: Neither blocks nor log segments found. Creating new log.
I20260807 18:29:17.242708 6835 log.cc:824] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: Log is configured to *not* fsync() on all Append() calls
I20260807 18:29:17.244598 6837 tablet_bootstrap.cc:654] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: Neither blocks nor log segments found. Creating new log.
I20260807 18:29:17.245914 6837 log.cc:824] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: Log is configured to *not* fsync() on all Append() calls
I20260807 18:29:17.246073 6835 tablet_bootstrap.cc:492] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: No bootstrap required, opened a new log
I20260807 18:29:17.246377 6835 ts_tablet_manager.cc:1397] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260807 18:29:17.246412 6836 tablet_bootstrap.cc:654] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: Neither blocks nor log segments found. Creating new log.
I20260807 18:29:17.247611 6836 log.cc:824] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: Log is configured to *not* fsync() on all Append() calls
I20260807 18:29:17.249020 6837 tablet_bootstrap.cc:492] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: No bootstrap required, opened a new log
I20260807 18:29:17.249320 6837 ts_tablet_manager.cc:1397] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260807 18:29:17.251346 6836 tablet_bootstrap.cc:492] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: No bootstrap required, opened a new log
I20260807 18:29:17.251598 6836 ts_tablet_manager.cc:1397] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260807 18:29:17.257174 6835 raft_consensus.cc:348] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } }
I20260807 18:29:17.257604 6835 raft_consensus.cc:374] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:29:17.257750 6835 raft_consensus.cc:729] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc6dfd4725a46559264faf38005abf9, State: Initialized, Role: FOLLOWER
I20260807 18:29:17.258348 6835 consensus_queue.cc:260] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } }
I20260807 18:29:17.259289 6836 raft_consensus.cc:348] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } }
I20260807 18:29:17.259629 6814 heartbeater.cc:503] Master 127.27.1.254:42909 was elected leader, sending a full tablet report...
I20260807 18:29:17.259843 6836 raft_consensus.cc:374] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:29:17.260028 6836 raft_consensus.cc:729] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc6db92d2af41ce941dd64a162e3a80, State: Initialized, Role: FOLLOWER
I20260807 18:29:17.259788 6837 raft_consensus.cc:348] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } }
I20260807 18:29:17.260313 6835 ts_tablet_manager.cc:1428] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260807 18:29:17.260315 6837 raft_consensus.cc:374] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:29:17.260414 6837 raft_consensus.cc:729] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5217e9952d48588bcd2c357d6fb4c0, State: Initialized, Role: FOLLOWER
I20260807 18:29:17.260740 6836 consensus_queue.cc:260] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } }
I20260807 18:29:17.261019 6837 consensus_queue.cc:260] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } }
I20260807 18:29:17.262459 6836 ts_tablet_manager.cc:1428] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260807 18:29:17.263700 6668 heartbeater.cc:503] Master 127.27.1.254:42909 was elected leader, sending a full tablet report...
I20260807 18:29:17.264230 6837 ts_tablet_manager.cc:1428] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
W20260807 18:29:17.337037 6524 tablet.cc:2369] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:29:17.377473 6669 tablet.cc:2369] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:29:17.387183 6815 tablet.cc:2369] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:29:17.541064 6843 raft_consensus.cc:482] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:29:17.541378 6843 raft_consensus.cc:504] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } }
I20260807 18:29:17.543139 6843 leader_election.cc:288] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dc6db92d2af41ce941dd64a162e3a80 (127.27.1.193:34469), 1bc6dfd4725a46559264faf38005abf9 (127.27.1.195:32889)
I20260807 18:29:17.550120 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.543528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:34165 (local address 127.27.1.193:34469)
0807 18:29:17.543710 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.543715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.544299 (+ 584us) server_negotiation.cc:407] Connection header received
0807 18:29:17.544573 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.544581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.544692 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.544772 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:29:17.545600 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.546313 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.548222 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.549067 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.549244 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.549321 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.549702 (+ 381us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.549857 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 18:29:17.550248 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.544502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:44285 (local address 127.27.1.195:32889)
0807 18:29:17.544701 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.544707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.544952 (+ 245us) server_negotiation.cc:407] Connection header received
0807 18:29:17.545066 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.545072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.545153 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.545243 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:29:17.545840 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.546561 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.548340 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.549191 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.549360 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.549440 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.549872 (+ 432us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.550015 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:29:17.550266 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.543531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34469 (local address 127.27.1.194:34165)
0807 18:29:17.544143 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:17.544184 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:17.544352 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:17.544981 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:17.544987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:17.545016 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:29:17.545458 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.545472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.546450 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.546455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.548056 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.548068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.549270 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.549276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.549454 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.549478 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 18:29:17.549668 (+ 190us) client_negotiation.cc:240] Negotiation successful
0807 18:29:17.549759 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":4,"thread_start_us":333,"threads_started":1}
I20260807 18:29:17.550406 6845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.544231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:32889 (local address 127.27.1.194:44285)
0807 18:29:17.544843 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:17.544857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:17.544958 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:17.545275 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:17.545278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:17.545296 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:29:17.545679 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.545687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.546706 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.546712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.548188 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.548201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.549409 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.549415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.549696 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.549728 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 18:29:17.549821 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 18:29:17.549933 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":23,"thread_start_us":307,"threads_started":1}
I20260807 18:29:17.551512 6762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a8da76a8754992a0e18b70e81ea178" candidate_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bc6dfd4725a46559264faf38005abf9" is_pre_election: true
I20260807 18:29:17.551555 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a8da76a8754992a0e18b70e81ea178" candidate_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc6db92d2af41ce941dd64a162e3a80" is_pre_election: true
I20260807 18:29:17.552122 6470 raft_consensus.cc:2454] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed5217e9952d48588bcd2c357d6fb4c0 in term 0.
I20260807 18:29:17.552136 6762 raft_consensus.cc:2454] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed5217e9952d48588bcd2c357d6fb4c0 in term 0.
I20260807 18:29:17.552371 6841 raft_consensus.cc:482] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:29:17.552659 6841 raft_consensus.cc:504] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } }
I20260807 18:29:17.552878 6548 leader_election.cc:302] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc6dfd4725a46559264faf38005abf9, ed5217e9952d48588bcd2c357d6fb4c0; no voters:
I20260807 18:29:17.553282 6843 raft_consensus.cc:2781] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:29:17.553431 6843 raft_consensus.cc:482] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:29:17.553547 6843 raft_consensus.cc:3037] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:29:17.554374 6841 leader_election.cc:288] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dc6db92d2af41ce941dd64a162e3a80 (127.27.1.193:34469), ed5217e9952d48588bcd2c357d6fb4c0 (127.27.1.194:44353)
I20260807 18:29:17.556659 6843 raft_consensus.cc:504] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } }
I20260807 18:29:17.557704 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a8da76a8754992a0e18b70e81ea178" candidate_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc6db92d2af41ce941dd64a162e3a80"
I20260807 18:29:17.557919 6470 raft_consensus.cc:3037] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:29:17.558020 6762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a8da76a8754992a0e18b70e81ea178" candidate_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bc6dfd4725a46559264faf38005abf9"
I20260807 18:29:17.558271 6762 raft_consensus.cc:3037] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:29:17.561205 6470 raft_consensus.cc:2454] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed5217e9952d48588bcd2c357d6fb4c0 in term 1.
I20260807 18:29:17.561578 6548 leader_election.cc:302] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc6db92d2af41ce941dd64a162e3a80, ed5217e9952d48588bcd2c357d6fb4c0; no voters:
I20260807 18:29:17.562155 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.554655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:44353 (local address 127.27.1.195:51475)
0807 18:29:17.554824 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:17.554843 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:17.554952 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:17.555700 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:17.555705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:17.555735 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:29:17.555901 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.555911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.557248 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.557253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.559480 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.559495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.561246 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.561253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.561492 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.561532 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 18:29:17.561675 (+ 143us) client_negotiation.cc:240] Negotiation successful
0807 18:29:17.561818 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":31}
I20260807 18:29:17.562127 6843 leader_election.cc:288] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1dc6db92d2af41ce941dd64a162e3a80 (127.27.1.193:34469), 1bc6dfd4725a46559264faf38005abf9 (127.27.1.195:32889)
I20260807 18:29:17.562239 6847 raft_consensus.cc:2781] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:29:17.562459 6847 raft_consensus.cc:686] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 1 LEADER]: Becoming Leader. State: Replica: ed5217e9952d48588bcd2c357d6fb4c0, State: Running, Role: LEADER
I20260807 18:29:17.562876 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.555201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:51475 (local address 127.27.1.194:44353)
0807 18:29:17.555391 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.555397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.555415 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:29:17.555494 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.555498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.555567 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.555656 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:29:17.556055 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.557109 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.559717 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.560900 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.561063 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.561146 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.562457 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.562640 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 18:29:17.563154 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.555247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:46497 (local address 127.27.1.193:34469)
0807 18:29:17.555846 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.555850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.555862 (+ 12us) server_negotiation.cc:407] Connection header received
0807 18:29:17.555903 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.555906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.555956 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.556056 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:29:17.556750 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.557784 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.560565 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.561926 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.562067 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.562150 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.562748 (+ 598us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.562869 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507}
I20260807 18:29:17.563170 6847 consensus_queue.cc:237] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } }
I20260807 18:29:17.562542 6762 raft_consensus.cc:2454] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed5217e9952d48588bcd2c357d6fb4c0 in term 1.
I20260807 18:29:17.563808 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.554703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34469 (local address 127.27.1.195:46497)
0807 18:29:17.555431 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0807 18:29:17.555448 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:29:17.555551 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:29:17.556207 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:29:17.556212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:29:17.556237 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:29:17.556585 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.556596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.557998 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.558008 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.560348 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:29:17.560362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.562343 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.562352 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:29:17.562552 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.562590 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 18:29:17.563308 (+ 718us) client_negotiation.cc:240] Negotiation successful
0807 18:29:17.563417 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":26,"thread_start_us":402,"threads_started":1}
I20260807 18:29:17.564422 6619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a8da76a8754992a0e18b70e81ea178" candidate_uuid: "1bc6dfd4725a46559264faf38005abf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" is_pre_election: true
I20260807 18:29:17.565506 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a8da76a8754992a0e18b70e81ea178" candidate_uuid: "1bc6dfd4725a46559264faf38005abf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc6db92d2af41ce941dd64a162e3a80" is_pre_election: true
I20260807 18:29:17.565835 6470 raft_consensus.cc:2379] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc6dfd4725a46559264faf38005abf9 in current term 1: Already voted for candidate ed5217e9952d48588bcd2c357d6fb4c0 in this term.
I20260807 18:29:17.566431 6695 leader_election.cc:302] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc6dfd4725a46559264faf38005abf9; no voters: 1dc6db92d2af41ce941dd64a162e3a80, ed5217e9952d48588bcd2c357d6fb4c0
I20260807 18:29:17.566788 6841 raft_consensus.cc:2726] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 18:29:17.568734 6308 catalog_manager.cc:5495] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed5217e9952d48588bcd2c357d6fb4c0 (127.27.1.194). New cstate: current_term: 1 leader_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed5217e9952d48588bcd2c357d6fb4c0" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44353 } health_report { overall_health: HEALTHY } } }
I20260807 18:29:17.669914 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.664419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.27.1.194:44353)
0807 18:29:17.664617 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.664623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.664728 (+ 105us) server_negotiation.cc:407] Connection header received
0807 18:29:17.664976 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.664980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.665069 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.665221 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:29:17.665590 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.666362 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.667758 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.668153 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.668289 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.668748 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:29:17.669228 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:29:17.669458 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.669511 (+ 53us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.669593 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:29:17.677783 6470 raft_consensus.cc:1261] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [term 1 FOLLOWER]: Refusing update from remote peer ed5217e9952d48588bcd2c357d6fb4c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:29:17.677824 6762 raft_consensus.cc:1261] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 1 FOLLOWER]: Refusing update from remote peer ed5217e9952d48588bcd2c357d6fb4c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:29:17.678656 6847 consensus_queue.cc:1035] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc6db92d2af41ce941dd64a162e3a80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:29:17.678966 6843 consensus_queue.cc:1035] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc6dfd4725a46559264faf38005abf9" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:29:17.689451 6850 mvcc.cc:205] Tried to move back new op lower bound from 7315977657038868480 to 7315977656584122368. Current Snapshot: MvccSnapshot[applied={T|T < 7315977657038868480}]
I20260807 18:29:17.710991 6308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:29:17.711339 6308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:29:17.714854 6308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2cb27ea4e2942f28095ff9454c4c45f: Sending DeleteTablet for 3 replicas of tablet a9a8da76a8754992a0e18b70e81ea178
I20260807 18:29:17.715942 6449 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a8da76a8754992a0e18b70e81ea178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:29:17 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:44276
I20260807 18:29:17.716022 6741 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a8da76a8754992a0e18b70e81ea178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:29:17 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:41450
I20260807 18:29:17.716234 6597 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a8da76a8754992a0e18b70e81ea178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:29:17 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:38948
I20260807 18:29:17.717622 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 6378
I20260807 18:29:17.733318 6860 tablet_replica.cc:330] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: stopping tablet replica
I20260807 18:29:17.733886 6860 raft_consensus.cc:2229] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:29:17.734539 6860 raft_consensus.cc:2258] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:29:17.735692 6862 tablet_replica.cc:330] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: stopping tablet replica
I20260807 18:29:17.736038 6862 raft_consensus.cc:2229] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:29:17.736550 6862 raft_consensus.cc:2258] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:29:17.741568 6860 ts_tablet_manager.cc:1905] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:29:17.743410 6862 ts_tablet_manager.cc:1905] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:29:17.751005 6862 ts_tablet_manager.cc:1918] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:29:17.751279 6862 log.cc:1196] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/a9a8da76a8754992a0e18b70e81ea178
I20260807 18:29:17.751945 6862 ts_tablet_manager.cc:1939] T a9a8da76a8754992a0e18b70e81ea178 P ed5217e9952d48588bcd2c357d6fb4c0: Deleting consensus metadata
I20260807 18:29:17.753882 6292 catalog_manager.cc:4841] TS ed5217e9952d48588bcd2c357d6fb4c0 (127.27.1.194:44353): tablet a9a8da76a8754992a0e18b70e81ea178 (table test-table [id=e62227e4dc5e4b30b810e3b55a3bbd95]) successfully deleted
Aug 07 18:29:17 dist-test-slave-1xrh krb5kdc[6261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:29:17.754626 6861 tablet_replica.cc:330] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: stopping tablet replica
I20260807 18:29:17.755084 6861 raft_consensus.cc:2229] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:29:17.756347 6861 raft_consensus.cc:2258] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:29:17.758553 6861 ts_tablet_manager.cc:1905] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:29:17.758975 6860 ts_tablet_manager.cc:1918] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:29:17.759224 6860 log.cc:1196] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/a9a8da76a8754992a0e18b70e81ea178
I20260807 18:29:17.759835 6860 ts_tablet_manager.cc:1939] T a9a8da76a8754992a0e18b70e81ea178 P 1dc6db92d2af41ce941dd64a162e3a80: Deleting consensus metadata
I20260807 18:29:17.761673 6294 catalog_manager.cc:4841] TS 1dc6db92d2af41ce941dd64a162e3a80 (127.27.1.193:34469): tablet a9a8da76a8754992a0e18b70e81ea178 (table test-table [id=e62227e4dc5e4b30b810e3b55a3bbd95]) successfully deleted
I20260807 18:29:17.763638 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.720012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44282 (local address 127.27.1.193:34469)
0807 18:29:17.720240 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.720249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.729814 (+ 9565us) server_negotiation.cc:407] Connection header received
0807 18:29:17.730377 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.730385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.730492 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.730583 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:17.735050 (+ 4467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.736541 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.747283 (+ 10742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.748203 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.748333 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.755562 (+ 7229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:17.755577 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:17.755595 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:17.755652 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:17.759953 (+ 4301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:17.761167 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:17.761176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:17.761179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:17.761338 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:17.761935 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:17.761940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:17.761943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:17.762392 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:17.762699 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.763022 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.763325 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 18:29:17.764256 6516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:17.766412 6861 ts_tablet_manager.cc:1918] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:29:17.766606 6861 log.cc:1196] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/a9a8da76a8754992a0e18b70e81ea178
I20260807 18:29:17.767139 6861 ts_tablet_manager.cc:1939] T a9a8da76a8754992a0e18b70e81ea178 P 1bc6dfd4725a46559264faf38005abf9: Deleting consensus metadata
I20260807 18:29:17.769237 6294 catalog_manager.cc:4841] TS 1bc6dfd4725a46559264faf38005abf9 (127.27.1.195:32889): tablet a9a8da76a8754992a0e18b70e81ea178 (table test-table [id=e62227e4dc5e4b30b810e3b55a3bbd95]) successfully deleted
I20260807 18:29:17.853241 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 6378
I20260807 18:29:17.876332 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 6529
Aug 07 18:29:17 dist-test-slave-1xrh krb5kdc[6261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:29:17.893251 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:17.877215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.27.1.194:44353)
0807 18:29:17.877447 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:17.877453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:17.877476 (+ 23us) server_negotiation.cc:407] Connection header received
0807 18:29:17.877664 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:17.877671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:17.877765 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:17.877902 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:17.879569 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.880418 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.881828 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:17.882277 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:17.882433 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:17.886869 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:17.886883 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:17.886899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:17.886932 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:17.890200 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:17.890906 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:17.890914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:17.890916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:17.891061 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:17.891513 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:17.891519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:17.891523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:17.891951 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:17.892335 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:17.892643 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 18:29:17.892911 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 18:29:17.894219 6663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:18.008474 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 6529
I20260807 18:29:18.031112 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 6674
Aug 07 18:29:18 dist-test-slave-1xrh krb5kdc[6261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:29:18.047045 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:29:18.031902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.27.1.195:32889)
0807 18:29:18.032083 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 18:29:18.032090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:29:18.032149 (+ 59us) server_negotiation.cc:407] Connection header received
0807 18:29:18.032359 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:29:18.032364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:29:18.032433 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:29:18.032547 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:29:18.033791 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:18.034541 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:18.035814 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:29:18.036215 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:29:18.036360 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:29:18.041429 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:29:18.041442 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:29:18.041458 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:29:18.041492 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:29:18.044318 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:18.044990 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:18.044997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:18.044999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:18.045120 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:29:18.045486 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:29:18.045492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:29:18.045495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:29:18.045867 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:29:18.046170 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0807 18:29:18.046459 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 18:29:18.046676 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 18:29:18.047725 6809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:18.138042 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 6674
I20260807 18:29:18.157637 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 6275
I20260807 18:29:18.158763 6345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:29:18.269133 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 6275
2026-08-07T18:29:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4758 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:29:18 dist-test-slave-1xrh krb5kdc[6870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:29:18 dist-test-slave-1xrh krb5kdc[6870](info): set up 2 sockets
Aug 07 18:29:18 dist-test-slave-1xrh krb5kdc[6870](info): commencing operation
krb5kdc: starting...
W20260807 18:29:19.442517 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:29:19 dist-test-slave-1xrh krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:29:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/rangeradmin_127.27.1.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/rangeradmin_127.27.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/rangerlookup_127.27.1.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/rangerlookup_127.27.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/HTTP_127.27.1.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/HTTP_127.27.1.212@KRBTEST.COM.keytab.
I20260807 18:29:19.821604 27655 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrFJVJ0/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/postgres -l logfile start
2026-08-07 18:29:22.422 UTC [6903] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 18:29:22.422 UTC [6903] LOG: listening on IPv4 address "127.27.1.212", port 57005
2026-08-07 18:29:22.426 UTC [6903] LOG: listening on Unix socket "/tmp/.s.PGSQL.57005"
2026-08-07 18:29:22.447 UTC [6907] LOG: database system was shut down at 2026-08-07 18:29:20 UTC
2026-08-07 18:29:22.451 UTC [6903] LOG: database system is ready to accept connections
I20260807 18:29:23.467911 27655 mini_postgres.cc:96] Postgres bound to 57005
2026-08-07 18:29:23.493 UTC [6915] FATAL: database "slave" does not exist
127.27.1.212:57005 - accepting connections
I20260807 18:29:23.494241 27655 mini_ranger.cc:161] Starting Ranger...
I20260807 18:29:23.525367 27655 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 18:29:24.028983 27655 mini_ranger.cc:87] Created ranger Postgres database
I20260807 18:29:24.029134 27655 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-admin
2026-08-07 18:29:24,374 [I] DB FLAVOR :POSTGRES
2026-08-07 18:29:24,374 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 18:29:24,374 [I] Checking connection..
2026-08-07 18:29:24,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 18:29:24,610 [I] Checking connection passed.
2026-08-07 18:29:24,610 [I] --------- Verifying version history table ---------
2026-08-07 18:29:24,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 18:29:24,868 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 18:29:24,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 18:29:25,114 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 18:29:25,115 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 18:29:25,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 18:29:25.342 UTC [6986] WARNING: there is no transaction in progress
2026-08-07 18:29:25,348 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 18:29:25,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 18:29:25,611 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 18:29:25,612 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 18:29:25,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 18:29:25,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 18:29:26,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1xrh.c.gcp-upstream.internal','N') ;"
2026-08-07 18:29:26,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 18:29:26,590 [I] Table x_portal_user does not exist in database ranger
2026-08-07 18:29:26,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 18:29:26,834 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 18:29:26,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 18:29:27,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 18:29:27,341 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 18:29:27,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 18:29:28.266 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.281 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.297 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.312 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.328 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.348 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.412 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.419 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.432 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.442 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.452 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.465 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.475 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.481 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.487 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:28.493 UTC [7103] WARNING: there is no transaction in progress
2026-08-07 18:29:29,132 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 18:29:29,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 18:29:29,391 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 18:29:29,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 18:29:29,630 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 18:29:29,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 18:29:29,887 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 18:29:29,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 18:29:30,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 18:29:30,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1xrh.c.gcp-upstream.internal';"
2026-08-07 18:29:30,598 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 18:29:30,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 18:29:30,839 [I] DB_PATCHES have already been applied
I20260807 18:29:30.845031 27655 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrFJVJ0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 18:29:30.845149 27655 mini_ranger.cc:193] Using host: 127.27.1.212
I20260807 18:29:30.866334 27655 mini_ranger.cc:227] Ranger admin URL: http://127.27.1.212:55071
Aug 07, 2026 6:29:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:29:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:29:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 6:29:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 6:29:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 6:29:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 6:29:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 18:29:32.072412 27655 mini_ranger.cc:160] Time spent starting Ranger: real 8.578s user 0.000s sys 0.060s
Aug 07, 2026 6:29:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 6:29:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 07, 2026 6:29:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 6:29:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 6:29:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 18:29:54.517840 27655 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/rangerkms_127.27.1.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/rangerkms_127.27.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.1.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.1.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.1.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/HTTP_127.27.1.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.1.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/HTTP_127.27.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 18:29:54 dist-test-slave-1xrh krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127394, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 18:29:54.713713 27655 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 18:29:54.739249 27655 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 18:29:55.644265 27655 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 18:29:55.644455 27655 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-kms
I20260807 18:29:55.644593 27655 mini_ranger_kms.cc:227] Using postgres at 127.27.1.212:57005
2026-08-07 18:29:56,146 [I] DB FLAVOR :POSTGRES
2026-08-07 18:29:56,146 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 18:29:56,146 [I] Checking connection
2026-08-07 18:29:56,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 18:29:56,398 [I] connection success
2026-08-07 18:29:56,399 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 18:29:56,399 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 18:29:56,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 18:29:56,627 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 18:29:56,627 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 18:29:56,628 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 18:29:56,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:57005/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 18:29:56,890 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 18:29:57.386641 27655 mini_ranger_kms.cc:310] Created kms service
I20260807 18:29:57.580117 27655 mini_ranger_kms.cc:326] Created kudu user
I20260807 18:29:57.625890 27655 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 18:29:57 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127394, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:29:58.010247 27655 mini_ranger_kms.cc:384] Added ranger policy
I20260807 18:29:58.011668 27655 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 18:29:58.011778 27655 mini_ranger_kms.cc:240] Using host: 127.27.1.212
I20260807 18:29:58.038759 27655 mini_ranger_kms.cc:276] Ranger KMS PID: 7299
I20260807 18:29:58.039086 27655 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.1.212:49793
Aug 07, 2026 6:29:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:29:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:29:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 6:29:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 6:29:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 6:29:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 18:30:00 dist-test-slave-1xrh krb5kdc[6870](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786127400, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.1.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:00.512181 27655 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.799s user 0.145s sys 0.397s
I20260807 18:30:00.512599 27655 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 18:30:00.512867 27655 mini_ranger_kms.cc:401] 127.27.1.212:49793/kms/v1/keys
Aug 07 18:30:00 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127400, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.1.212@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
Aug 07 18:30:01 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127400, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.1.212@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
Aug 07 18:30:03 dist-test-slave-1xrh krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:30:03.452972 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35793
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:35793
--ranger_config_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:49793/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:30:03.811048 7352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:30:03.811345 7352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:30:03.811538 7352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:30:03.818704 7352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:30:03.818809 7352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:30:03.818856 7352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:30:03.818917 7352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:30:03.818966 7352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:30:03.829026 7352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.1.212:49793/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:35793
--ranger_config_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:30:03.830240 7352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:30:03.831550 7352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:30:03.845445 7357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:03.845494 7358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:03.845593 7360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:03.846194 7352 server_base.cc:1034] running on GCE node
I20260807 18:30:03.846992 7352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:30:03.848377 7352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:30:03.849586 7352 hybrid_clock.cc:648] HybridClock initialized: now 1786127403849539 us; error 30 us; skew 500 ppm
I20260807 18:30:03.849922 7352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:30:03 dist-test-slave-1xrh krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127403, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:03.860016 7352 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:30:03.861698 7352 webserver.cc:460] Webserver started at http://127.27.1.254:43525/ using document root <none> and password file <none>
I20260807 18:30:03.862382 7352 fs_manager.cc:359] Metadata directory not provided
I20260807 18:30:03.862519 7352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:30:03.862857 7352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 18:30:03 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127403, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:03.920960 7352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "bb758912713a4b3aa4d6e57a92db4e38"
format_stamp: "Formatted at 2026-08-07 18:30:03 on dist-test-slave-1xrh"
server_key: "cbe7ee1c38af86d256c030dce396a432"
server_key_iv: "d00503fc5d9227ab455dbd957781710d"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:03.922106 7352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "bb758912713a4b3aa4d6e57a92db4e38"
format_stamp: "Formatted at 2026-08-07 18:30:03 on dist-test-slave-1xrh"
server_key: "cbe7ee1c38af86d256c030dce396a432"
server_key_iv: "d00503fc5d9227ab455dbd957781710d"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:03.929548 7352 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260807 18:30:03.951719 7367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:03.953395 7352 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.004s
I20260807 18:30:03.953600 7352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "bb758912713a4b3aa4d6e57a92db4e38"
format_stamp: "Formatted at 2026-08-07 18:30:03 on dist-test-slave-1xrh"
server_key: "cbe7ee1c38af86d256c030dce396a432"
server_key_iv: "d00503fc5d9227ab455dbd957781710d"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:03.953842 7352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:30:03.972370 7352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:30:03.976639 7352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:30:03.976958 7352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:30:03.997491 7352 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:35793
I20260807 18:30:03.997538 7424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:35793 every 8 connection(s)
I20260807 18:30:03.998901 7352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:30:04.002769 7425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:30:04.005239 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 7352
I20260807 18:30:04.005467 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:30:04.006325 27655 external_mini_cluster.cc:1428] Setting key e1cdc4361285acf87cea1af6c9bc8e18
I20260807 18:30:04.014883 7425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38: Bootstrap starting.
I20260807 18:30:04.018081 7425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38: Neither blocks nor log segments found. Creating new log.
I20260807 18:30:04.019093 7425 log.cc:824] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38: Log is configured to *not* fsync() on all Append() calls
Aug 07 18:30:04 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:30:04.022408 7425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38: No bootstrap required, opened a new log
I20260807 18:30:04.028832 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:04.008651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.27.1.254:35793)
0807 18:30:04.009502 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:04.009514 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:04.009556 (+ 42us) server_negotiation.cc:407] Connection header received
0807 18:30:04.011288 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:04.011311 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:04.011696 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:04.012099 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:04.013422 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:04.014707 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:04.016133 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:04.016834 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:04.017020 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:04.021588 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:04.021595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:04.021609 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:04.021643 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:04.024828 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:04.025646 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:04.025652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:04.025657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:04.025812 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:04.026202 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:04.026206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:04.026207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:04.026686 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:04.026932 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:04.027432 (+ 500us) server_negotiation.cc:299] Negotiation successful
0807 18:30:04.027785 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":425,"threads_started":1}
I20260807 18:30:04.031751 7425 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb758912713a4b3aa4d6e57a92db4e38" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35793 } }
I20260807 18:30:04.032114 7425 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:30:04.032219 7425 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb758912713a4b3aa4d6e57a92db4e38, State: Initialized, Role: FOLLOWER
I20260807 18:30:04.032733 7425 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb758912713a4b3aa4d6e57a92db4e38" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35793 } }
I20260807 18:30:04.032982 7425 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:30:04.033170 7425 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:30:04.033387 7425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:30:04.035918 7425 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb758912713a4b3aa4d6e57a92db4e38" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35793 } }
I20260807 18:30:04.036403 7425 leader_election.cc:302] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb758912713a4b3aa4d6e57a92db4e38; no voters:
I20260807 18:30:04.036978 7425 leader_election.cc:288] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:30:04.037101 7431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:30:04.037909 7431 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [term 1 LEADER]: Becoming Leader. State: Replica: bb758912713a4b3aa4d6e57a92db4e38, State: Running, Role: LEADER
I20260807 18:30:04.038664 7431 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb758912713a4b3aa4d6e57a92db4e38" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35793 } }
I20260807 18:30:04.040189 7425 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:30:04.042003 7432 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb758912713a4b3aa4d6e57a92db4e38. Latest consensus state: current_term: 1 leader_uuid: "bb758912713a4b3aa4d6e57a92db4e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb758912713a4b3aa4d6e57a92db4e38" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35793 } } }
I20260807 18:30:04.042085 7433 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb758912713a4b3aa4d6e57a92db4e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb758912713a4b3aa4d6e57a92db4e38" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35793 } } }
I20260807 18:30:04.042382 7432 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [sys.catalog]: This master's current role is: LEADER
I20260807 18:30:04.042505 7433 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38 [sys.catalog]: This master's current role is: LEADER
I20260807 18:30:04.048823 7436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:30:04.049122 7425 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 18:30:04.061651 7436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:30:04.077111 7436 catalog_manager.cc:1347] Generated new cluster ID: 7fbbdcde358943c281841ed0e15049bc
I20260807 18:30:04.077307 7436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:30:04.089017 7436 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:30:04.090397 7436 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:30:04.103575 7436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38: Generated new TSK 0
I20260807 18:30:04.104555 7436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:30:04.901211 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:49793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35793
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:30:05 dist-test-slave-1xrh krb5kdc[6870](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786127405, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 18:30:05.364362 7473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:30:05.364765 7473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:30:05.364966 7473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:30:05.377100 7473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:30:05.377264 7473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:30:05.377528 7473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:30:05.397336 7473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.1.212:49793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:30:05.399580 7473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:30:05.401656 7473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:30:05.433785 7486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:05.439723 7485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:05.441372 7488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:05.444792 7473 server_base.cc:1034] running on GCE node
I20260807 18:30:05.445833 7473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:30:05.447752 7473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:30:05.451315 7473 hybrid_clock.cc:648] HybridClock initialized: now 1786127405451094 us; error 124 us; skew 500 ppm
I20260807 18:30:05.451891 7473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:30:05 dist-test-slave-1xrh krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127405, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:05.458630 7473 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:30:05.461387 7473 webserver.cc:460] Webserver started at http://127.27.1.193:35205/ using document root <none> and password file <none>
I20260807 18:30:05.462167 7473 fs_manager.cc:359] Metadata directory not provided
I20260807 18:30:05.462385 7473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:30:05.462843 7473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 18:30:05 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127405, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:05.483744 7473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "7198d8919dbd41f8aad4291867785019"
format_stamp: "Formatted at 2026-08-07 18:30:05 on dist-test-slave-1xrh"
server_key: "5f1018d8c4c2f5598f996c6567aa4e77"
server_key_iv: "063e6a7d2e5b64e758cd9fa3e37b521a"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:05.484854 7473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "7198d8919dbd41f8aad4291867785019"
format_stamp: "Formatted at 2026-08-07 18:30:05 on dist-test-slave-1xrh"
server_key: "5f1018d8c4c2f5598f996c6567aa4e77"
server_key_iv: "063e6a7d2e5b64e758cd9fa3e37b521a"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:05.493883 7473 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260807 18:30:05.514428 7495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:05.516444 7473 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260807 18:30:05.516716 7473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "7198d8919dbd41f8aad4291867785019"
format_stamp: "Formatted at 2026-08-07 18:30:05 on dist-test-slave-1xrh"
server_key: "5f1018d8c4c2f5598f996c6567aa4e77"
server_key_iv: "063e6a7d2e5b64e758cd9fa3e37b521a"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:05.517079 7473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:30:05.537737 7473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:30:05.554177 7473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:30:05.555024 7473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:30:05.563361 7473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:30:05.573028 7473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:30:05.573277 7473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 18:30:05.573654 7473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:30:05.573928 7473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 18:30:05 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127405, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:06.069659 7473 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:38479
I20260807 18:30:06.069828 7610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:38479 every 8 connection(s)
I20260807 18:30:06.072095 7473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:30:06.080092 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 7473
I20260807 18:30:06.080324 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:30:06.081372 27655 external_mini_cluster.cc:1428] Setting key 753a32f2eee8df73a5b3464f4d80645d
Aug 07 18:30:06 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127405, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:30:06.121763 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:06.077907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:57945 (local address 127.27.1.254:35793)
0807 18:30:06.088115 (+ 10208us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:06.088122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:06.094169 (+ 6047us) server_negotiation.cc:407] Connection header received
0807 18:30:06.094262 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:06.094268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:06.094433 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:06.094600 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:06.098649 (+ 4049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:06.100461 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:06.103901 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:06.104910 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:06.105034 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:06.111963 (+ 6929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:06.111971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:06.111977 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:06.112022 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:06.116838 (+ 4816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:06.117944 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:06.117950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:06.117953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:06.118237 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:06.119003 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:06.119008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:06.119011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:06.119571 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:06.119703 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:06.120848 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0807 18:30:06.121123 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10006,"thread_start_us":9771,"threads_started":1}
I20260807 18:30:06.125432 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:06.077155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35793 (local address 127.27.1.193:57945)
0807 18:30:06.088844 (+ 11689us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:06.088907 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:06.093604 (+ 4697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:06.095203 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:06.095218 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:06.096547 (+ 1329us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:30:06.098230 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:06.098261 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:06.100756 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:06.100761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:06.103654 (+ 2893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:06.103710 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:06.105275 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:06.105285 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:06.105692 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:06.106835 (+ 1143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:30:06.106867 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:30:06.111535 (+ 4668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:30:06.117016 (+ 5481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:06.117026 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:06.117045 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:06.117805 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:06.118437 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:06.118441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:06.118444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:06.118869 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:06.119796 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:30:06.119803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:30:06.120342 (+ 539us) client_negotiation.cc:769] Sending connection context
0807 18:30:06.122695 (+ 2353us) client_negotiation.cc:240] Negotiation successful
0807 18:30:06.123381 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11304,"thread_start_us":21947,"threads_started":1}
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
I20260807 18:30:06.127732 7611 heartbeater.cc:347] Connected to a master server at 127.27.1.254:35793
I20260807 18:30:06.128188 7611 heartbeater.cc:464] Registering TS with master...
I20260807 18:30:06.129501 7611 heartbeater.cc:511] Master 127.27.1.254:35793 requested a full tablet report, sending...
I20260807 18:30:06.132591 7384 ts_manager.cc:194] Registered new tserver with Master: 7198d8919dbd41f8aad4291867785019 (127.27.1.193:38479)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
I20260807 18:30:06.136061 7384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:57945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:30:06.179771 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:49793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35793
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:30:06 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127405, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
W20260807 18:30:06.686368 7619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:30:06.686713 7619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:30:06.686892 7619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:30:06.694480 7619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:30:06.694633 7619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:30:06.694820 7619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:30:06.705246 7619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.1.212:49793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:30:06.706655 7619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:30:06.708112 7619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:30:06.724272 7637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:06.724664 7633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:06.724272 7635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:06.727077 7619 server_base.cc:1034] running on GCE node
I20260807 18:30:06.727912 7619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:30:06.729199 7619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:30:06.730499 7619 hybrid_clock.cc:648] HybridClock initialized: now 1786127406730378 us; error 54 us; skew 500 ppm
I20260807 18:30:06.730954 7619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:30:06 dist-test-slave-1xrh krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:06.734817 7619 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:30:06.736562 7619 webserver.cc:460] Webserver started at http://127.27.1.194:42169/ using document root <none> and password file <none>
I20260807 18:30:06.737283 7619 fs_manager.cc:359] Metadata directory not provided
I20260807 18:30:06.737419 7619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:30:06.737767 7619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 18:30:06 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:06.761144 7619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "73fb575f281249598dc3382c572f1c25"
format_stamp: "Formatted at 2026-08-07 18:30:06 on dist-test-slave-1xrh"
server_key: "33bf55f9aec682b9f6580102d0fda2dd"
server_key_iv: "247457c396e913076a3f93fe88462977"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:06.762432 7619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "73fb575f281249598dc3382c572f1c25"
format_stamp: "Formatted at 2026-08-07 18:30:06 on dist-test-slave-1xrh"
server_key: "33bf55f9aec682b9f6580102d0fda2dd"
server_key_iv: "247457c396e913076a3f93fe88462977"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:06.770503 7619 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.004s
I20260807 18:30:06.797844 7644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:06.800307 7619 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.010s sys 0.000s
I20260807 18:30:06.800568 7619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "73fb575f281249598dc3382c572f1c25"
format_stamp: "Formatted at 2026-08-07 18:30:06 on dist-test-slave-1xrh"
server_key: "33bf55f9aec682b9f6580102d0fda2dd"
server_key_iv: "247457c396e913076a3f93fe88462977"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:06.800910 7619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:30:06.832589 7619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:30:06.838443 7619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:30:06.838804 7619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:30:06.840166 7619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:30:06.842293 7619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:30:06.842410 7619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:06.842540 7619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:30:06.842605 7619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:06.900915 7619 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:35801
I20260807 18:30:06.900969 7767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:35801 every 8 connection(s)
I20260807 18:30:06.902628 7619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:30:06.911167 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 7619
I20260807 18:30:06.911371 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:30:06.911811 27655 external_mini_cluster.cc:1428] Setting key 19957fd384eca893dc722b28fad788f7
Aug 07 18:30:06 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:30:06.927696 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:06.905634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:48245 (local address 127.27.1.254:35793)
0807 18:30:06.906203 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:06.906210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:06.907279 (+ 1069us) server_negotiation.cc:407] Connection header received
0807 18:30:06.909186 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:06.909193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:06.909285 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:06.909426 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:06.910957 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:06.912253 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:06.914542 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:06.915218 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:06.915365 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:06.920113 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:06.920120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:06.920126 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:06.920170 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:06.924139 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:06.925111 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:06.925118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:06.925122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:06.925292 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:06.925848 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:06.925853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:06.925857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:06.926156 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:06.926318 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:06.927034 (+ 716us) server_negotiation.cc:299] Negotiation successful
0807 18:30:06.927306 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":293,"threads_started":1}
I20260807 18:30:06.928747 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:06.906157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35793 (local address 127.27.1.194:48245)
0807 18:30:06.907076 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:06.907148 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:06.908914 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:06.909623 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:06.909631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:06.910166 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:30:06.910744 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:06.910771 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:06.912484 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:06.912489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:06.914352 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:06.914379 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:06.915696 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:06.915704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:06.916034 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:06.916845 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:30:06.916875 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:30:06.919847 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:30:06.924291 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:06.924303 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:06.924323 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:06.924965 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:06.925460 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:06.925467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:06.925474 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:06.925719 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:06.926301 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:30:06.926309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:30:06.926642 (+ 333us) client_negotiation.cc:769] Sending connection context
0807 18:30:06.926924 (+ 282us) client_negotiation.cc:240] Negotiation successful
0807 18:30:06.927499 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":418,"threads_started":1}
I20260807 18:30:06.930244 7769 heartbeater.cc:347] Connected to a master server at 127.27.1.254:35793
I20260807 18:30:06.930576 7769 heartbeater.cc:464] Registering TS with master...
I20260807 18:30:06.931370 7769 heartbeater.cc:511] Master 127.27.1.254:35793 requested a full tablet report, sending...
I20260807 18:30:06.932981 7384 ts_manager.cc:194] Registered new tserver with Master: 73fb575f281249598dc3382c572f1c25 (127.27.1.194:35801)
I20260807 18:30:06.933884 7384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:48245
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:30:06.986956 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:49793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35793
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:30:07.140272 7611 heartbeater.cc:503] Master 127.27.1.254:35793 was elected leader, sending a full tablet report...
W20260807 18:30:07.383682 7775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:30:07.383993 7775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:30:07.384097 7775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:30:07.391834 7775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:30:07.391932 7775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:30:07.392042 7775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:30:07.401552 7775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.1.212:49793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:30:07.402802 7775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:30:07.404155 7775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:30:07.417507 7780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:07.417533 7782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:07.417744 7784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:07.418879 7775 server_base.cc:1034] running on GCE node
I20260807 18:30:07.419536 7775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:30:07.420472 7775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:30:07.421732 7775 hybrid_clock.cc:648] HybridClock initialized: now 1786127407421673 us; error 55 us; skew 500 ppm
I20260807 18:30:07.422106 7775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:30:07 dist-test-slave-1xrh krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127407, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:07.425402 7775 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:30:07.426997 7775 webserver.cc:460] Webserver started at http://127.27.1.195:35841/ using document root <none> and password file <none>
I20260807 18:30:07.427832 7775 fs_manager.cc:359] Metadata directory not provided
I20260807 18:30:07.427995 7775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:30:07.428326 7775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 18:30:07 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127407, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:07.445225 7775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "ebfddacf3430415e8ef7cd1683b73a6b"
format_stamp: "Formatted at 2026-08-07 18:30:07 on dist-test-slave-1xrh"
server_key: "6b9489d0d7977243bb7832f496f00316"
server_key_iv: "429fe218e300b3d8ba6386f80455d1f7"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:07.446398 7775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "ebfddacf3430415e8ef7cd1683b73a6b"
format_stamp: "Formatted at 2026-08-07 18:30:07 on dist-test-slave-1xrh"
server_key: "6b9489d0d7977243bb7832f496f00316"
server_key_iv: "429fe218e300b3d8ba6386f80455d1f7"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:07.453635 7775 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260807 18:30:07.470256 7791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:07.471851 7775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:30:07.472038 7775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "ebfddacf3430415e8ef7cd1683b73a6b"
format_stamp: "Formatted at 2026-08-07 18:30:07 on dist-test-slave-1xrh"
server_key: "6b9489d0d7977243bb7832f496f00316"
server_key_iv: "429fe218e300b3d8ba6386f80455d1f7"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:07.472275 7775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:30:07.485620 7775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:30:07.489365 7775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:30:07.489616 7775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:30:07.490454 7775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:30:07.492143 7775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:30:07.492218 7775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:07.492336 7775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:30:07.492408 7775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:07.528672 7775 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:44327
I20260807 18:30:07.528709 7913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:44327 every 8 connection(s)
I20260807 18:30:07.529982 7775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:30:07.539433 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 7775
I20260807 18:30:07.539613 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:30:07.540186 27655 external_mini_cluster.cc:1428] Setting key 41bea3fafdbd5869915218debcda293c
Aug 07 18:30:07 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127407, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:30:07.550945 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.532511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:48059 (local address 127.27.1.254:35793)
0807 18:30:07.533003 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:07.533009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:07.534120 (+ 1111us) server_negotiation.cc:407] Connection header received
0807 18:30:07.536020 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:07.536025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:07.536110 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:07.536208 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:07.537655 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.538378 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.540650 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.541212 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.541388 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.545395 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:07.545400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:07.545403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:07.545449 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:07.548215 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:07.548912 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:07.548916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:07.548918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:07.549048 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:07.549451 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:07.549455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:07.549457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:07.549711 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:07.549820 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:07.550520 (+ 700us) server_negotiation.cc:299] Negotiation successful
0807 18:30:07.550685 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":237,"threads_started":1}
I20260807 18:30:07.551880 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.533118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35793 (local address 127.27.1.195:48059)
0807 18:30:07.533950 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:07.534000 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:07.535743 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:07.536399 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:07.536407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:07.536821 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:30:07.537498 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.537518 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.538511 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.538514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.540494 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.540510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.541510 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.541523 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.541918 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.542765 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:30:07.542786 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:30:07.545185 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:30:07.548339 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:07.548345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:07.548356 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:07.548804 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:07.549146 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:07.549149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:07.549151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:07.549341 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:07.549816 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:30:07.549821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:30:07.550140 (+ 319us) client_negotiation.cc:769] Sending connection context
0807 18:30:07.550356 (+ 216us) client_negotiation.cc:240] Negotiation successful
0807 18:30:07.550721 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":312,"threads_started":1}
I20260807 18:30:07.553251 7914 heartbeater.cc:347] Connected to a master server at 127.27.1.254:35793
I20260807 18:30:07.553540 7914 heartbeater.cc:464] Registering TS with master...
I20260807 18:30:07.554394 7914 heartbeater.cc:511] Master 127.27.1.254:35793 requested a full tablet report, sending...
I20260807 18:30:07.556237 7383 ts_manager.cc:194] Registered new tserver with Master: ebfddacf3430415e8ef7cd1683b73a6b (127.27.1.195:44327)
I20260807 18:30:07.557266 7383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:48059
I20260807 18:30:07.568188 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:30:07.585338 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.572976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.27.1.254:35793)
0807 18:30:07.573169 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:07.573173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:07.573507 (+ 334us) server_negotiation.cc:407] Connection header received
0807 18:30:07.573646 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:07.573649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:07.573723 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:07.573803 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:07.575245 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.576000 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.577586 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.577967 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.578055 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.580436 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:07.580441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:07.580444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:07.580474 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:07.583048 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:07.583605 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:07.583608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:07.583611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:07.583792 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:07.584129 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:07.584132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:07.584134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:07.584381 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:07.584560 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:07.584836 (+ 276us) server_negotiation.cc:299] Negotiation successful
0807 18:30:07.585012 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:30:07.589520 7383 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:30:07.596748 7383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:30:07.632330 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.609652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.27.1.193:38479)
0807 18:30:07.622813 (+ 13161us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:07.622821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:07.622853 (+ 32us) server_negotiation.cc:407] Connection header received
0807 18:30:07.622961 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:07.622969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:07.623222 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:07.623496 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:30:07.624468 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.626275 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.629396 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.630878 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.631024 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.631288 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:07.631788 (+ 500us) server_negotiation.cc:299] Negotiation successful
0807 18:30:07.631959 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12989,"thread_start_us":12839,"threads_started":1}
I20260807 18:30:07.632781 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.609547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:38479 (local address 127.0.0.1:40410)
0807 18:30:07.619770 (+ 10223us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:07.619827 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:07.620035 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:07.623501 (+ 3466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:07.623506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:07.623526 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:30:07.624289 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.624305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.626417 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.626422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.629233 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.629248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.631113 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.631146 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.631426 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.631582 (+ 156us) client_negotiation.cc:769] Sending connection context
0807 18:30:07.631713 (+ 131us) client_negotiation.cc:240] Negotiation successful
0807 18:30:07.631993 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9912,"mutex_wait_us":391,"thread_start_us":11486,"threads_started":1}
I20260807 18:30:07.633173 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.609837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:44327 (local address 127.0.0.1:52540)
0807 18:30:07.621346 (+ 11509us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:07.621367 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:07.621509 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:07.623996 (+ 2487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:07.624001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:07.624034 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:30:07.624690 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.624704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.626834 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.626843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.629108 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.629121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.631158 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.631163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.631430 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.631582 (+ 152us) client_negotiation.cc:769] Sending connection context
0807 18:30:07.631713 (+ 131us) client_negotiation.cc:240] Negotiation successful
0807 18:30:07.631983 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11356,"mutex_wait_us":324,"thread_start_us":11928,"threads_started":1}
I20260807 18:30:07.634053 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.610113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:35801 (local address 127.0.0.1:50156)
0807 18:30:07.622060 (+ 11947us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:07.622088 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:07.622216 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:07.623240 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:07.623248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:07.623282 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:30:07.624056 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.624074 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.626169 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.626176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.629233 (+ 3057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.629248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.633209 (+ 3961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.633215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.633428 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.633467 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 18:30:07.633561 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 18:30:07.633688 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11794,"mutex_wait_us":310,"thread_start_us":10139,"threads_started":1}
I20260807 18:30:07.634094 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.611281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.27.1.194:35801)
0807 18:30:07.619028 (+ 7747us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:07.619042 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:07.622343 (+ 3301us) server_negotiation.cc:407] Connection header received
0807 18:30:07.622548 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:07.622563 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:07.622892 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:07.623089 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:30:07.624743 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.626036 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.631382 (+ 5346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.632694 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.632840 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.633021 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:07.633706 (+ 685us) server_negotiation.cc:299] Negotiation successful
0807 18:30:07.633835 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7497,"thread_start_us":7357,"threads_started":1}
I20260807 18:30:07.635010 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.610562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.27.1.195:44327)
0807 18:30:07.623283 (+ 12721us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:07.623291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:07.623326 (+ 35us) server_negotiation.cc:407] Connection header received
0807 18:30:07.623426 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:07.623435 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:07.623649 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:07.624011 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:30:07.625290 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.626691 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.629270 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.630878 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.631021 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.631292 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:07.632668 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0807 18:30:07.634625 (+ 1957us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12520,"thread_start_us":12296,"threads_started":1}
I20260807 18:30:07.637035 7544 tablet_service.cc:1467] Processing CreateTablet for tablet c49449b6a086475193d1bcddf4a96c4d (DEFAULT_TABLE table=test-table [id=54eef079257a44099a51627be74b1315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:30:07.637025 7696 tablet_service.cc:1467] Processing CreateTablet for tablet c49449b6a086475193d1bcddf4a96c4d (DEFAULT_TABLE table=test-table [id=54eef079257a44099a51627be74b1315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:30:07.638370 7696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49449b6a086475193d1bcddf4a96c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:30:07.638375 7544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49449b6a086475193d1bcddf4a96c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:30:07.641790 7843 tablet_service.cc:1467] Processing CreateTablet for tablet c49449b6a086475193d1bcddf4a96c4d (DEFAULT_TABLE table=test-table [id=54eef079257a44099a51627be74b1315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:30:07.643209 7843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49449b6a086475193d1bcddf4a96c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:30:07.650204 7935 tablet_bootstrap.cc:492] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: Bootstrap starting.
I20260807 18:30:07.654078 7935 tablet_bootstrap.cc:654] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: Neither blocks nor log segments found. Creating new log.
I20260807 18:30:07.655261 7935 log.cc:824] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: Log is configured to *not* fsync() on all Append() calls
I20260807 18:30:07.656082 7936 tablet_bootstrap.cc:492] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: Bootstrap starting.
I20260807 18:30:07.659145 7939 tablet_bootstrap.cc:492] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: Bootstrap starting.
I20260807 18:30:07.659889 7935 tablet_bootstrap.cc:492] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: No bootstrap required, opened a new log
I20260807 18:30:07.659952 7936 tablet_bootstrap.cc:654] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: Neither blocks nor log segments found. Creating new log.
I20260807 18:30:07.660164 7935 ts_tablet_manager.cc:1397] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 18:30:07.661231 7936 log.cc:824] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: Log is configured to *not* fsync() on all Append() calls
I20260807 18:30:07.662657 7939 tablet_bootstrap.cc:654] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: Neither blocks nor log segments found. Creating new log.
I20260807 18:30:07.663677 7939 log.cc:824] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: Log is configured to *not* fsync() on all Append() calls
I20260807 18:30:07.664532 7936 tablet_bootstrap.cc:492] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: No bootstrap required, opened a new log
I20260807 18:30:07.664793 7936 ts_tablet_manager.cc:1397] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 18:30:07.666100 7939 tablet_bootstrap.cc:492] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: No bootstrap required, opened a new log
I20260807 18:30:07.666363 7939 ts_tablet_manager.cc:1397] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 18:30:07.671265 7935 raft_consensus.cc:348] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 } } peers { permanent_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44327 } } peers { permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 } }
I20260807 18:30:07.672047 7935 raft_consensus.cc:374] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:30:07.672155 7935 raft_consensus.cc:729] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7198d8919dbd41f8aad4291867785019, State: Initialized, Role: FOLLOWER
I20260807 18:30:07.672793 7935 consensus_queue.cc:260] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 } } peers { permanent_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44327 } } peers { permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 } }
I20260807 18:30:07.674656 7935 ts_tablet_manager.cc:1428] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260807 18:30:07.674577 7939 raft_consensus.cc:348] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 } } peers { permanent_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44327 } } peers { permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 } }
I20260807 18:30:07.674973 7939 raft_consensus.cc:374] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:30:07.675091 7939 raft_consensus.cc:729] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfddacf3430415e8ef7cd1683b73a6b, State: Initialized, Role: FOLLOWER
I20260807 18:30:07.675832 7939 consensus_queue.cc:260] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 } } peers { permanent_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44327 } } peers { permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 } }
I20260807 18:30:07.676414 7936 raft_consensus.cc:348] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 } } peers { permanent_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44327 } } peers { permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 } }
I20260807 18:30:07.676904 7936 raft_consensus.cc:374] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:30:07.677002 7936 raft_consensus.cc:729] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73fb575f281249598dc3382c572f1c25, State: Initialized, Role: FOLLOWER
I20260807 18:30:07.677141 7914 heartbeater.cc:503] Master 127.27.1.254:35793 was elected leader, sending a full tablet report...
I20260807 18:30:07.677584 7936 consensus_queue.cc:260] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 } } peers { permanent_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44327 } } peers { permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 } }
I20260807 18:30:07.677816 7939 ts_tablet_manager.cc:1428] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 18:30:07.679023 7769 heartbeater.cc:503] Master 127.27.1.254:35793 was elected leader, sending a full tablet report...
I20260807 18:30:07.679726 7936 ts_tablet_manager.cc:1428] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: Time spent starting tablet: real 0.015s user 0.006s sys 0.009s
W20260807 18:30:07.781808 7915 tablet.cc:2369] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:30:07.824406 7943 raft_consensus.cc:482] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:30:07.824692 7943 raft_consensus.cc:504] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 } } peers { permanent_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44327 } } peers { permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 } }
I20260807 18:30:07.826610 7943 leader_election.cc:288] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7198d8919dbd41f8aad4291867785019 (127.27.1.193:38479), 73fb575f281249598dc3382c572f1c25 (127.27.1.194:35801)
I20260807 18:30:07.833266 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.827051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:38479 (local address 127.27.1.195:51303)
0807 18:30:07.827534 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:07.827559 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:07.827728 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:07.828179 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:07.828184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:07.828213 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:30:07.828485 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.828498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.829662 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.829667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.831330 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.831342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.832502 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.832507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.832711 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.832776 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 18:30:07.832863 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 18:30:07.832953 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":79}
I20260807 18:30:07.833385 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.827053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:51303 (local address 127.27.1.193:38479)
0807 18:30:07.827324 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:07.827332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:07.827658 (+ 326us) server_negotiation.cc:407] Connection header received
0807 18:30:07.827952 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:07.827961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:07.828052 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:07.828141 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:30:07.828665 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.829532 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.831488 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.832349 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.832513 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.832567 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:07.832934 (+ 367us) server_negotiation.cc:299] Negotiation successful
0807 18:30:07.833080 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 18:30:07.833570 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.827057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:52495 (local address 127.27.1.194:35801)
0807 18:30:07.827312 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:07.827322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:07.828020 (+ 698us) server_negotiation.cc:407] Connection header received
0807 18:30:07.828189 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:07.828194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:07.828270 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:07.828393 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:30:07.829059 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.829864 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.831707 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.832521 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.832671 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.832740 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:07.833082 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 18:30:07.833207 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 18:30:07.833753 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.827177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:35801 (local address 127.27.1.195:52495)
0807 18:30:07.827902 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:07.827919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:07.828048 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:07.828435 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:07.828438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:07.828458 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:30:07.828882 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.828893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.830019 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.830022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.831548 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:07.831565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.832725 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.832730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:07.832910 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.832943 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 18:30:07.833032 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 18:30:07.833142 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":3,"thread_start_us":378,"threads_started":1}
I20260807 18:30:07.834793 7717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49449b6a086475193d1bcddf4a96c4d" candidate_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73fb575f281249598dc3382c572f1c25" is_pre_election: true
I20260807 18:30:07.834798 7564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49449b6a086475193d1bcddf4a96c4d" candidate_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7198d8919dbd41f8aad4291867785019" is_pre_election: true
I20260807 18:30:07.835345 7717 raft_consensus.cc:2454] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfddacf3430415e8ef7cd1683b73a6b in term 0.
I20260807 18:30:07.835492 7564 raft_consensus.cc:2454] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfddacf3430415e8ef7cd1683b73a6b in term 0.
W20260807 18:30:07.835599 7612 tablet.cc:2369] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:30:07.836131 7792 leader_election.cc:302] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73fb575f281249598dc3382c572f1c25, ebfddacf3430415e8ef7cd1683b73a6b; no voters:
I20260807 18:30:07.836553 7943 raft_consensus.cc:2781] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:30:07.836714 7943 raft_consensus.cc:482] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:30:07.836841 7943 raft_consensus.cc:3037] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:30:07.839044 7943 raft_consensus.cc:504] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 } } peers { permanent_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44327 } } peers { permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 } }
I20260807 18:30:07.839690 7943 leader_election.cc:288] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [CANDIDATE]: Term 1 election: Requested vote from peers 7198d8919dbd41f8aad4291867785019 (127.27.1.193:38479), 73fb575f281249598dc3382c572f1c25 (127.27.1.194:35801)
I20260807 18:30:07.840209 7717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49449b6a086475193d1bcddf4a96c4d" candidate_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73fb575f281249598dc3382c572f1c25"
I20260807 18:30:07.840209 7564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49449b6a086475193d1bcddf4a96c4d" candidate_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7198d8919dbd41f8aad4291867785019"
I20260807 18:30:07.840413 7717 raft_consensus.cc:3037] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:30:07.840413 7564 raft_consensus.cc:3037] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:30:07.843724 7564 raft_consensus.cc:2454] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfddacf3430415e8ef7cd1683b73a6b in term 1.
I20260807 18:30:07.843724 7717 raft_consensus.cc:2454] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfddacf3430415e8ef7cd1683b73a6b in term 1.
I20260807 18:30:07.844152 7795 leader_election.cc:302] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7198d8919dbd41f8aad4291867785019, 73fb575f281249598dc3382c572f1c25, ebfddacf3430415e8ef7cd1683b73a6b; no voters:
I20260807 18:30:07.844504 7943 raft_consensus.cc:2781] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:30:07.844987 7943 raft_consensus.cc:686] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 1 LEADER]: Becoming Leader. State: Replica: ebfddacf3430415e8ef7cd1683b73a6b, State: Running, Role: LEADER
I20260807 18:30:07.845567 7943 consensus_queue.cc:237] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 } } peers { permanent_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44327 } } peers { permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 } }
I20260807 18:30:07.850049 7383 catalog_manager.cc:5495] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b reported cstate change: term changed from 0 to 1, leader changed from <none> to ebfddacf3430415e8ef7cd1683b73a6b (127.27.1.195). New cstate: current_term: 1 leader_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebfddacf3430415e8ef7cd1683b73a6b" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 } health_report { overall_health: UNKNOWN } } }
I20260807 18:30:07.868759 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.863423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.27.1.195:44327)
0807 18:30:07.863663 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:07.863668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:07.863689 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:30:07.863887 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:07.863891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:07.863974 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:07.864091 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:30:07.864424 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.865131 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.866713 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.867088 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.867235 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.867692 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:30:07.868155 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:30:07.868262 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:07.868404 (+ 142us) server_negotiation.cc:299] Negotiation successful
0807 18:30:07.868502 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 18:30:07.876869 7564 raft_consensus.cc:1261] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019 [term 1 FOLLOWER]: Refusing update from remote peer ebfddacf3430415e8ef7cd1683b73a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:30:07.876951 7717 raft_consensus.cc:1261] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25 [term 1 FOLLOWER]: Refusing update from remote peer ebfddacf3430415e8ef7cd1683b73a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:30:07.877830 7946 consensus_queue.cc:1035] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7198d8919dbd41f8aad4291867785019" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 38479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:30:07.878221 7943 consensus_queue.cc:1035] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "73fb575f281249598dc3382c572f1c25" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 35801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:30:07.894254 7951 mvcc.cc:205] Tried to move back new op lower bound from 7315977862653333504 to 7315977862539460608. Current Snapshot: MvccSnapshot[applied={T|T < 7315977862653333504}]
I20260807 18:30:07.904175 7384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:30:07.904428 7384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:30:07.908049 7384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb758912713a4b3aa4d6e57a92db4e38: Sending DeleteTablet for 3 replicas of tablet c49449b6a086475193d1bcddf4a96c4d
I20260807 18:30:07.909155 7544 tablet_service.cc:1514] Processing DeleteTablet for tablet c49449b6a086475193d1bcddf4a96c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:30:07 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:40410
I20260807 18:30:07.909165 7843 tablet_service.cc:1514] Processing DeleteTablet for tablet c49449b6a086475193d1bcddf4a96c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:30:07 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:52540
I20260807 18:30:07.912796 7696 tablet_service.cc:1514] Processing DeleteTablet for tablet c49449b6a086475193d1bcddf4a96c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:30:07 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:50156
I20260807 18:30:07.915722 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 7473
I20260807 18:30:07.924293 7962 tablet_replica.cc:330] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: stopping tablet replica
I20260807 18:30:07.924818 7962 raft_consensus.cc:2229] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:30:07.925580 7962 raft_consensus.cc:2258] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:30:07.930581 7962 ts_tablet_manager.cc:1905] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:30:07.930610 7961 tablet_replica.cc:330] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: stopping tablet replica
I20260807 18:30:07.931105 7961 raft_consensus.cc:2229] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:30:07.931748 7961 raft_consensus.cc:2258] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:30:07.932377 7963 tablet_replica.cc:330] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: stopping tablet replica
I20260807 18:30:07.932861 7963 raft_consensus.cc:2229] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:30:07.933382 7963 raft_consensus.cc:2258] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:30:07.935525 7963 ts_tablet_manager.cc:1905] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:30:07.935853 7961 ts_tablet_manager.cc:1905] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:30:07.938227 7962 ts_tablet_manager.cc:1918] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:30:07.938416 7962 log.cc:1196] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/c49449b6a086475193d1bcddf4a96c4d
I20260807 18:30:07.938978 7962 ts_tablet_manager.cc:1939] T c49449b6a086475193d1bcddf4a96c4d P ebfddacf3430415e8ef7cd1683b73a6b: Deleting consensus metadata
I20260807 18:30:07.940902 7370 catalog_manager.cc:4841] TS ebfddacf3430415e8ef7cd1683b73a6b (127.27.1.195:44327): tablet c49449b6a086475193d1bcddf4a96c4d (table test-table [id=54eef079257a44099a51627be74b1315]) successfully deleted
I20260807 18:30:07.942881 7961 ts_tablet_manager.cc:1918] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:30:07.943048 7961 log.cc:1196] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/c49449b6a086475193d1bcddf4a96c4d
I20260807 18:30:07.943657 7961 ts_tablet_manager.cc:1939] T c49449b6a086475193d1bcddf4a96c4d P 7198d8919dbd41f8aad4291867785019: Deleting consensus metadata
I20260807 18:30:07.943794 7963 ts_tablet_manager.cc:1918] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:30:07.944021 7963 log.cc:1196] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/c49449b6a086475193d1bcddf4a96c4d
I20260807 18:30:07.944615 7963 ts_tablet_manager.cc:1939] T c49449b6a086475193d1bcddf4a96c4d P 73fb575f281249598dc3382c572f1c25: Deleting consensus metadata
Aug 07 18:30:07 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:30:07.945596 7371 catalog_manager.cc:4841] TS 7198d8919dbd41f8aad4291867785019 (127.27.1.193:38479): tablet c49449b6a086475193d1bcddf4a96c4d (table test-table [id=54eef079257a44099a51627be74b1315]) successfully deleted
I20260807 18:30:07.946527 7368 catalog_manager.cc:4841] TS 73fb575f281249598dc3382c572f1c25 (127.27.1.194:35801): tablet c49449b6a086475193d1bcddf4a96c4d (table test-table [id=54eef079257a44099a51627be74b1315]) successfully deleted
I20260807 18:30:07.953052 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:07.917419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.27.1.193:38479)
0807 18:30:07.917581 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:07.917588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:07.919017 (+ 1429us) server_negotiation.cc:407] Connection header received
0807 18:30:07.919926 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:07.919935 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:07.920038 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:07.920131 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:07.926118 (+ 5987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.927565 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.932188 (+ 4623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:07.932853 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:07.932970 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:07.946590 (+ 13620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:07.946605 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:07.946623 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:07.946665 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:07.949693 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:07.950630 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:07.950637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:07.950639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:07.950760 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:07.951235 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:07.951239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:07.951241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:07.951589 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:07.951871 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:07.952329 (+ 458us) server_negotiation.cc:299] Negotiation successful
0807 18:30:07.952600 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 18:30:07.953657 7606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:30:08.048223 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 7473
I20260807 18:30:08.069528 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 7619
Aug 07 18:30:08 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:30:08.087769 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:08.070473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50160 (local address 127.27.1.194:35801)
0807 18:30:08.070673 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:08.070680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:08.070699 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:30:08.070838 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:08.070843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:08.070932 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:08.071055 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:08.072500 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:08.073697 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:08.074997 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:08.075789 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:08.075930 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:08.080620 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:08.080640 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:08.080663 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:08.080711 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:08.084658 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:08.085340 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:08.085350 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:08.085354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:08.085516 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:08.086044 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:08.086050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:08.086057 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:08.086456 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:08.086787 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:08.087107 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 18:30:08.087389 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 18:30:08.088562 7762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:30:08.182399 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 7619
I20260807 18:30:08.207924 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 7775
Aug 07 18:30:08 dist-test-slave-1xrh krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:30:08.226033 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:08.208775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.27.1.195:44327)
0807 18:30:08.208947 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:08.208952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:08.209073 (+ 121us) server_negotiation.cc:407] Connection header received
0807 18:30:08.209296 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:08.209306 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:08.209376 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:08.209518 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:08.210831 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:08.211720 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:08.213037 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:08.213557 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:08.213710 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:08.218562 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:08.218579 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:08.218601 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:08.218664 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:08.222906 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:08.223712 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:08.223721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:08.223725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:08.223927 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:08.224321 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:08.224326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:08.224330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:08.224836 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:08.225175 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:08.225395 (+ 220us) server_negotiation.cc:299] Negotiation successful
0807 18:30:08.225645 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:30:08.226881 7908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:30:08.352886 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 7775
I20260807 18:30:08.376647 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 7352
I20260807 18:30:08.377988 7419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:30:08.492893 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 7352
I20260807 18:30:08.523277 27655 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 18:30:08.868534 27655 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 18:30:08.868811 27655 mini_ranger.cc:66] Stopping Ranger...
I20260807 18:30:09.214066 27655 mini_ranger.cc:68] Stopped Ranger
2026-08-07 18:30:09.214 UTC [6903] LOG: received smart shutdown request
2026-08-07 18:30:09.218 UTC [6903] LOG: background worker "logical replication launcher" (PID 6913) exited with exit code 1
2026-08-07 18:30:09.219 UTC [6908] LOG: shutting down
2026-08-07 18:30:09.396 UTC [6903] LOG: database system is shut down
2026-08-07T18:30:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51253 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:30:09 dist-test-slave-1xrh krb5kdc[7977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:30:09 dist-test-slave-1xrh krb5kdc[7977](info): set up 2 sockets
Aug 07 18:30:09 dist-test-slave-1xrh krb5kdc[7977](info): commencing operation
krb5kdc: starting...
W20260807 18:30:10.713379 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:30:10 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:30:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:30:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/rangeradmin_127.27.1.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/rangeradmin_127.27.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/rangerlookup_127.27.1.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/rangerlookup_127.27.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/HTTP_127.27.1.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/HTTP_127.27.1.212@KRBTEST.COM.keytab.
I20260807 18:30:11.093528 27655 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrFJVJ0/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/postgres -l logfile start
2026-08-07 18:30:13.602 UTC [8008] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 18:30:13.602 UTC [8008] LOG: listening on IPv4 address "127.27.1.212", port 53689
2026-08-07 18:30:13.605 UTC [8008] LOG: listening on Unix socket "/tmp/.s.PGSQL.53689"
2026-08-07 18:30:13.628 UTC [8011] LOG: database system was shut down at 2026-08-07 18:30:12 UTC
2026-08-07 18:30:13.632 UTC [8008] LOG: database system is ready to accept connections
I20260807 18:30:14.636324 27655 mini_postgres.cc:96] Postgres bound to 53689
2026-08-07 18:30:14.658 UTC [8020] FATAL: database "slave" does not exist
127.27.1.212:53689 - accepting connections
I20260807 18:30:14.659559 27655 mini_ranger.cc:161] Starting Ranger...
I20260807 18:30:14.688900 27655 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 18:30:15.152098 27655 mini_ranger.cc:87] Created ranger Postgres database
I20260807 18:30:15.152236 27655 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-admin
2026-08-07 18:30:15,463 [I] DB FLAVOR :POSTGRES
2026-08-07 18:30:15,463 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 18:30:15,464 [I] Checking connection..
2026-08-07 18:30:15,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 18:30:15,712 [I] Checking connection passed.
2026-08-07 18:30:15,712 [I] --------- Verifying version history table ---------
2026-08-07 18:30:15,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 18:30:15,986 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 18:30:15,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 18:30:16,233 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 18:30:16,234 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 18:30:16,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 18:30:16.467 UTC [8089] WARNING: there is no transaction in progress
2026-08-07 18:30:16,474 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 18:30:16,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 18:30:16,708 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 18:30:16,708 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 18:30:16,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 18:30:16,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 18:30:17,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1xrh.c.gcp-upstream.internal','N') ;"
2026-08-07 18:30:17,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 18:30:17,741 [I] Table x_portal_user does not exist in database ranger
2026-08-07 18:30:17,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 18:30:17,970 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 18:30:17,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 18:30:18,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 18:30:18,450 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 18:30:18,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 18:30:19.400 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.416 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.432 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.447 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.466 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.482 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.535 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.542 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.555 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.564 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.574 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.587 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.596 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.602 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.608 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:19.614 UTC [8209] WARNING: there is no transaction in progress
2026-08-07 18:30:20,227 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 18:30:20,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 18:30:20,452 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 18:30:20,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 18:30:20,695 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 18:30:20,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 18:30:20,946 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 18:30:20,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 18:30:21,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 18:30:21,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1xrh.c.gcp-upstream.internal';"
2026-08-07 18:30:21,709 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 18:30:21,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 18:30:21,934 [I] DB_PATCHES have already been applied
I20260807 18:30:21.940474 27655 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrFJVJ0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 18:30:21.940559 27655 mini_ranger.cc:193] Using host: 127.27.1.212
I20260807 18:30:21.960894 27655 mini_ranger.cc:227] Ranger admin URL: http://127.27.1.212:36045
Aug 07, 2026 6:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 6:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 6:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 6:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 6:30:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 18:30:23.220546 27655 mini_ranger.cc:160] Time spent starting Ranger: real 8.561s user 0.003s sys 0.054s
Aug 07, 2026 6:30:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 6:30:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 07, 2026 6:30:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 6:30:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 6:30:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 18:30:45.092351 27655 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/rangerkms_127.27.1.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/rangerkms_127.27.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.1.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.1.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.1.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/HTTP_127.27.1.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.1.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/HTTP_127.27.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 18:30:45 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127445, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 18:30:45.288004 27655 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 18:30:45.313859 27655 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 18:30:46.231319 27655 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 18:30:46.231451 27655 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-kms
I20260807 18:30:46.231508 27655 mini_ranger_kms.cc:227] Using postgres at 127.27.1.212:53689
2026-08-07 18:30:46,602 [I] DB FLAVOR :POSTGRES
2026-08-07 18:30:46,603 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 18:30:46,603 [I] Checking connection
2026-08-07 18:30:46,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 18:30:46,834 [I] connection success
2026-08-07 18:30:46,834 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 18:30:46,834 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 18:30:46,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 18:30:47,061 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 18:30:47,062 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 18:30:47,062 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 18:30:47,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.1.212:53689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 18:30:47,350 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 18:30:47.784355 27655 mini_ranger_kms.cc:310] Created kms service
I20260807 18:30:48.010596 27655 mini_ranger_kms.cc:326] Created kudu user
I20260807 18:30:48.055817 27655 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 18:30:48 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127445, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:48.465670 27655 mini_ranger_kms.cc:384] Added ranger policy
I20260807 18:30:48.465986 27655 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrFJVJ0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrFJVJ0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 18:30:48.466028 27655 mini_ranger_kms.cc:240] Using host: 127.27.1.212
I20260807 18:30:48.491808 27655 mini_ranger_kms.cc:276] Ranger KMS PID: 8406
I20260807 18:30:48.491951 27655 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.1.212:54915
Aug 07, 2026 6:30:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:30:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:30:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 6:30:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 6:30:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 6:30:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260807 18:30:50.923301 27655 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.635s user 0.047s sys 0.355s
I20260807 18:30:50.923568 27655 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 18:30:50.923769 27655 mini_ranger_kms.cc:401] 127.27.1.212:54915/kms/v1/keys
Aug 07 18:30:51 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786127451, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.1.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 18:30:51 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127451, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.1.212@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
Aug 07 18:30:51 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127451, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.1.212@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
Aug 07 18:30:54 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:30:54.303273 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35579
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:35579
--ranger_config_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:54915/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:30:54.645843 8460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:30:54.646122 8460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:30:54.646308 8460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:30:54.653584 8460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:30:54.653684 8460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:30:54.653810 8460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:30:54.653905 8460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:30:54.653990 8460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:30:54.664126 8460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.1.212:54915/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:35579
--ranger_config_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:30:54.665328 8460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:30:54.666698 8460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:30:54.678860 8466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:54.679051 8465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:54.678982 8468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:54.680037 8460 server_base.cc:1034] running on GCE node
I20260807 18:30:54.680795 8460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:30:54.682197 8460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:30:54.683393 8460 hybrid_clock.cc:648] HybridClock initialized: now 1786127454683355 us; error 36 us; skew 500 ppm
I20260807 18:30:54.683734 8460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:30:54 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127454, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:54.686748 8460 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:30:54.688272 8460 webserver.cc:460] Webserver started at http://127.27.1.254:39271/ using document root <none> and password file <none>
I20260807 18:30:54.688977 8460 fs_manager.cc:359] Metadata directory not provided
I20260807 18:30:54.689117 8460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:30:54.689427 8460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 18:30:54 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127454, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:54.744215 8460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "df5f30b0378d4095b8844766dd816d5f"
format_stamp: "Formatted at 2026-08-07 18:30:54 on dist-test-slave-1xrh"
server_key: "e5c44676bcb046d17573c4b36e8ed4fa"
server_key_iv: "b0ac90021f82e4738f2cf7f074b39cf6"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:54.745105 8460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "df5f30b0378d4095b8844766dd816d5f"
format_stamp: "Formatted at 2026-08-07 18:30:54 on dist-test-slave-1xrh"
server_key: "e5c44676bcb046d17573c4b36e8ed4fa"
server_key_iv: "b0ac90021f82e4738f2cf7f074b39cf6"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:54.750682 8460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 18:30:54.769770 8476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:54.771458 8460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 18:30:54.771643 8460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "df5f30b0378d4095b8844766dd816d5f"
format_stamp: "Formatted at 2026-08-07 18:30:54 on dist-test-slave-1xrh"
server_key: "e5c44676bcb046d17573c4b36e8ed4fa"
server_key_iv: "b0ac90021f82e4738f2cf7f074b39cf6"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:54.771871 8460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:30:54.805637 8460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:30:54.809484 8460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:30:54.809823 8460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:30:54.829011 8460 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:35579
I20260807 18:30:54.829041 8528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:35579 every 8 connection(s)
I20260807 18:30:54.830403 8460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:30:54.831284 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 8460
I20260807 18:30:54.831583 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:30:54.832329 27655 external_mini_cluster.cc:1428] Setting key cfee6c5c969a6cfb5f59ee9944a4fed0
I20260807 18:30:54.835304 8529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 18:30:54 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:30:54.849172 8529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: Bootstrap starting.
I20260807 18:30:54.853962 8529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: Neither blocks nor log segments found. Creating new log.
I20260807 18:30:54.855283 8529 log.cc:824] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: Log is configured to *not* fsync() on all Append() calls
I20260807 18:30:54.858893 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:54.834458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.27.1.254:35579)
0807 18:30:54.835334 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:54.835352 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:54.835407 (+ 55us) server_negotiation.cc:407] Connection header received
0807 18:30:54.837927 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:54.837956 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:54.838478 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:54.838935 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:54.840167 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:54.842521 (+ 2354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:54.844068 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:54.845186 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:54.845414 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:54.849683 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:54.849695 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:54.849716 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:54.849769 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:54.854060 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:54.854724 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:54.854733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:54.854743 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:54.854958 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:54.855437 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:54.855442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:54.855446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:54.856207 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:54.856480 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:54.857141 (+ 661us) server_negotiation.cc:299] Negotiation successful
0807 18:30:54.857558 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":365,"threads_started":1}
I20260807 18:30:54.858917 8529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: No bootstrap required, opened a new log
I20260807 18:30:54.867201 8529 raft_consensus.cc:348] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } }
I20260807 18:30:54.867544 8529 raft_consensus.cc:374] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:30:54.867623 8529 raft_consensus.cc:729] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5f30b0378d4095b8844766dd816d5f, State: Initialized, Role: FOLLOWER
I20260807 18:30:54.868160 8529 consensus_queue.cc:260] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } }
I20260807 18:30:54.868362 8529 raft_consensus.cc:388] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:30:54.868456 8529 raft_consensus.cc:482] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:30:54.868582 8529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:30:54.870569 8529 raft_consensus.cc:504] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } }
I20260807 18:30:54.870995 8529 leader_election.cc:302] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df5f30b0378d4095b8844766dd816d5f; no voters:
I20260807 18:30:54.871587 8529 leader_election.cc:288] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:30:54.871718 8535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:30:54.873113 8535 raft_consensus.cc:686] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 1 LEADER]: Becoming Leader. State: Replica: df5f30b0378d4095b8844766dd816d5f, State: Running, Role: LEADER
I20260807 18:30:54.873662 8535 consensus_queue.cc:237] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } }
I20260807 18:30:54.873862 8529 sys_catalog.cc:564] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:30:54.876904 8538 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader df5f30b0378d4095b8844766dd816d5f. Latest consensus state: current_term: 1 leader_uuid: "df5f30b0378d4095b8844766dd816d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } } }
I20260807 18:30:54.877280 8538 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [sys.catalog]: This master's current role is: LEADER
I20260807 18:30:54.877867 8536 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df5f30b0378d4095b8844766dd816d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } } }
I20260807 18:30:54.878237 8536 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [sys.catalog]: This master's current role is: LEADER
I20260807 18:30:54.878459 8544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:30:54.886622 8544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:30:54.887439 8529 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 18:30:54.907747 8544 catalog_manager.cc:1347] Generated new cluster ID: c9e526a96e2f4951a09f869f5b4105ca
I20260807 18:30:54.907900 8544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:30:54.935647 8544 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:30:54.937760 8544 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:30:54.948740 8544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: Generated new TSK 0
I20260807 18:30:54.949791 8544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:30:55.691171 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:54915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35579
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:30:55 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786127455, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 18:30:56.104887 8580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:30:56.105203 8580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:30:56.105371 8580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:30:56.113698 8580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:30:56.113844 8580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:30:56.114029 8580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:30:56.124729 8580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.1.212:54915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:30:56.126077 8580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:30:56.127472 8580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:30:56.144778 8591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:56.144874 8592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:56.148089 8594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:56.149046 8580 server_base.cc:1034] running on GCE node
I20260807 18:30:56.149878 8580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:30:56.150956 8580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:30:56.152261 8580 hybrid_clock.cc:648] HybridClock initialized: now 1786127456152201 us; error 46 us; skew 500 ppm
I20260807 18:30:56.152661 8580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:30:56 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127456, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:56.156625 8580 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:30:56.158248 8580 webserver.cc:460] Webserver started at http://127.27.1.193:33075/ using document root <none> and password file <none>
I20260807 18:30:56.159157 8580 fs_manager.cc:359] Metadata directory not provided
I20260807 18:30:56.159305 8580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:30:56.159639 8580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 18:30:56 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127456, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:56.179935 8580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "db2572262af44aa29dd509c132933c57"
format_stamp: "Formatted at 2026-08-07 18:30:56 on dist-test-slave-1xrh"
server_key: "eef70df10641151d8b7fb97da960bc78"
server_key_iv: "50dee5d5a58e88490e5d4484f0dd18c3"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:56.181646 8580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "db2572262af44aa29dd509c132933c57"
format_stamp: "Formatted at 2026-08-07 18:30:56 on dist-test-slave-1xrh"
server_key: "eef70df10641151d8b7fb97da960bc78"
server_key_iv: "50dee5d5a58e88490e5d4484f0dd18c3"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:56.201397 8580 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.010s sys 0.007s
I20260807 18:30:56.232890 8602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:56.240569 8580 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.002s sys 0.008s
I20260807 18:30:56.240819 8580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "db2572262af44aa29dd509c132933c57"
format_stamp: "Formatted at 2026-08-07 18:30:56 on dist-test-slave-1xrh"
server_key: "eef70df10641151d8b7fb97da960bc78"
server_key_iv: "50dee5d5a58e88490e5d4484f0dd18c3"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:56.241083 8580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:30:56.301420 8580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:30:56.306448 8580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:30:56.306815 8580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:30:56.308050 8580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:30:56.309892 8580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:30:56.309985 8580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:56.310102 8580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:30:56.310165 8580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:56.378749 8580 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:40565
I20260807 18:30:56.378791 8723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:40565 every 8 connection(s)
I20260807 18:30:56.380569 8580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:30:56.388211 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 8580
I20260807 18:30:56.388485 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:30:56.389320 27655 external_mini_cluster.cc:1428] Setting key c4dd27db2c6b3f37a1559357834a9652
Aug 07 18:30:56 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127456, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
I20260807 18:30:56.442761 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:56.383522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:55343 (local address 127.27.1.254:35579)
0807 18:30:56.414655 (+ 31133us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:56.414665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:56.419535 (+ 4870us) server_negotiation.cc:407] Connection header received
0807 18:30:56.422943 (+ 3408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:56.422950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:56.423060 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:56.423169 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:56.425544 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:56.426872 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:56.429491 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:56.430161 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:56.430310 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:56.435003 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:56.435010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:56.435017 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:56.435061 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:56.438754 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:56.439788 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:56.439794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:56.439798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:56.439966 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:56.440569 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:56.440574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:56.440577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:56.440909 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:56.441053 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:56.442103 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0807 18:30:56.442366 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30959,"thread_start_us":35855,"threads_started":1}
I20260807 18:30:56.444022 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:56.384367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35579 (local address 127.27.1.193:55343)
0807 18:30:56.415355 (+ 30988us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:56.415432 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:56.421953 (+ 6521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:56.423758 (+ 1805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:56.423770 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:56.424516 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:30:56.425321 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:56.425347 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:56.427095 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:56.427101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:56.429270 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:56.429287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:56.430648 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:56.430656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:56.430946 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:56.431831 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:30:56.431864 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:30:56.434717 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:30:56.438938 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:56.438948 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:56.438966 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:56.439632 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:56.440108 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:56.440113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:56.440117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:56.440426 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:56.441144 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:30:56.441152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:30:56.441596 (+ 444us) client_negotiation.cc:769] Sending connection context
0807 18:30:56.441935 (+ 339us) client_negotiation.cc:240] Negotiation successful
0807 18:30:56.442516 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30578,"thread_start_us":34634,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
I20260807 18:30:56.448374 8724 heartbeater.cc:347] Connected to a master server at 127.27.1.254:35579
I20260807 18:30:56.448755 8724 heartbeater.cc:464] Registering TS with master...
I20260807 18:30:56.449740 8724 heartbeater.cc:511] Master 127.27.1.254:35579 requested a full tablet report, sending...
I20260807 18:30:56.452579 8493 ts_manager.cc:194] Registered new tserver with Master: db2572262af44aa29dd509c132933c57 (127.27.1.193:40565)
I20260807 18:30:56.454623 8493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:55343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
W20260807 18:30:56.495256 27655 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.1.194: real 0.103s user 0.006s sys 0.019s
I20260807 18:30:56.496526 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:54915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35579
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:30:56 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127455, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
Aug 07 18:30:56 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127455, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
W20260807 18:30:57.022910 8730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:30:57.023437 8730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:30:57.023631 8730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:30:57.034950 8730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:30:57.035161 8730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:30:57.035378 8730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:30:57.050688 8730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.1.212:54915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:30:57.052582 8730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:30:57.054387 8730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:30:57.091708 8737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:57.093858 8738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:57.100371 8741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:57.100806 8730 server_base.cc:1034] running on GCE node
I20260807 18:30:57.101709 8730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:30:57.106506 8730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:30:57.107774 8730 hybrid_clock.cc:648] HybridClock initialized: now 1786127457107678 us; error 99 us; skew 500 ppm
I20260807 18:30:57.108242 8730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:30:57 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127457, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:57.119539 8730 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:30:57.150106 8730 webserver.cc:460] Webserver started at http://127.27.1.194:43235/ using document root <none> and password file <none>
I20260807 18:30:57.151443 8730 fs_manager.cc:359] Metadata directory not provided
I20260807 18:30:57.151687 8730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:30:57.152128 8730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 18:30:57 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127457, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:57.194931 8730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "75e79053c5894d988ae6a0340818f0f2"
format_stamp: "Formatted at 2026-08-07 18:30:57 on dist-test-slave-1xrh"
server_key: "f27f6e842724a5ba23391eca0ad42eef"
server_key_iv: "5c9aae6cb6c717247c8a531ba59e3084"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:57.196416 8730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "75e79053c5894d988ae6a0340818f0f2"
format_stamp: "Formatted at 2026-08-07 18:30:57 on dist-test-slave-1xrh"
server_key: "f27f6e842724a5ba23391eca0ad42eef"
server_key_iv: "5c9aae6cb6c717247c8a531ba59e3084"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:57.229679 8730 fs_manager.cc:691] Time spent creating directory manager: real 0.033s user 0.015s sys 0.016s
I20260807 18:30:57.271839 8754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:57.274013 8730 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.005s sys 0.013s
I20260807 18:30:57.274199 8730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "75e79053c5894d988ae6a0340818f0f2"
format_stamp: "Formatted at 2026-08-07 18:30:57 on dist-test-slave-1xrh"
server_key: "f27f6e842724a5ba23391eca0ad42eef"
server_key_iv: "5c9aae6cb6c717247c8a531ba59e3084"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:57.274529 8730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:30:57.320061 8730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:30:57.325569 8730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:30:57.325894 8730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:30:57.327214 8730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:30:57.329069 8730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:30:57.329165 8730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:57.329346 8730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:30:57.329413 8730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:57.371852 8730 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:45219
I20260807 18:30:57.371902 8885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:45219 every 8 connection(s)
I20260807 18:30:57.373533 8730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:30:57.382767 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 8730
I20260807 18:30:57.382946 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:30:57.383638 27655 external_mini_cluster.cc:1428] Setting key d85544ae0d0e8f90091334e020fe04c5
Aug 07 18:30:57 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127457, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:30:57.412775 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:57.379487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:56705 (local address 127.27.1.254:35579)
0807 18:30:57.387481 (+ 7994us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:57.387487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:57.387509 (+ 22us) server_negotiation.cc:407] Connection header received
0807 18:30:57.387575 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:57.387580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:57.387664 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:57.387788 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:57.390023 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:57.391298 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:57.399263 (+ 7965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:57.399989 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:57.400127 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:57.404820 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:57.404828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:57.404833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:57.404882 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:57.408700 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:57.409690 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:57.409696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:57.409700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:57.409884 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:57.410612 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:57.410618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:57.410621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:57.411028 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:57.411214 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:57.412133 (+ 919us) server_negotiation.cc:299] Negotiation successful
0807 18:30:57.412379 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7864,"thread_start_us":12433,"threads_started":1}
I20260807 18:30:57.413995 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:57.377830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35579 (local address 127.27.1.194:56705)
0807 18:30:57.383711 (+ 5881us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:57.383776 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:57.386617 (+ 2841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:57.388052 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:57.388064 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:57.388803 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:30:57.389673 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:57.389699 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:57.391482 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:57.391488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:57.393616 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:57.393631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:57.400446 (+ 6815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:57.400454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:57.400751 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:57.401645 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:30:57.401677 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:30:57.404541 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:30:57.408870 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:57.408880 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:57.408899 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:57.409529 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:57.410137 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:57.410143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:57.410147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:57.410449 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:57.411203 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:30:57.411212 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:30:57.411669 (+ 457us) client_negotiation.cc:769] Sending connection context
0807 18:30:57.412005 (+ 336us) client_negotiation.cc:240] Negotiation successful
0807 18:30:57.412579 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5565,"thread_start_us":13191,"threads_started":1}
I20260807 18:30:57.416018 8886 heartbeater.cc:347] Connected to a master server at 127.27.1.254:35579
I20260807 18:30:57.416368 8886 heartbeater.cc:464] Registering TS with master...
I20260807 18:30:57.417303 8886 heartbeater.cc:511] Master 127.27.1.254:35579 requested a full tablet report, sending...
I20260807 18:30:57.419148 8493 ts_manager.cc:194] Registered new tserver with Master: 75e79053c5894d988ae6a0340818f0f2 (127.27.1.194:45219)
I20260807 18:30:57.420926 8493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:56705
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
I20260807 18:30:57.458460 8724 heartbeater.cc:503] Master 127.27.1.254:35579 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:30:57.475622 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:54915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35579
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:30:57.860787 8892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:30:57.861064 8892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:30:57.861159 8892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:30:57.868160 8892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:30:57.868259 8892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:30:57.868367 8892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:30:57.878101 8892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.1.212:54915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:30:57.879392 8892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:30:57.880915 8892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:30:57.894161 8897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:57.894133 8898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:57.894228 8901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:57.895358 8892 server_base.cc:1034] running on GCE node
I20260807 18:30:57.895941 8892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:30:57.896831 8892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:30:57.898030 8892 hybrid_clock.cc:648] HybridClock initialized: now 1786127457897993 us; error 27 us; skew 500 ppm
I20260807 18:30:57.898301 8892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:30:57 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127457, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:57.901393 8892 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:30:57.902851 8892 webserver.cc:460] Webserver started at http://127.27.1.195:39415/ using document root <none> and password file <none>
I20260807 18:30:57.903591 8892 fs_manager.cc:359] Metadata directory not provided
I20260807 18:30:57.903731 8892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:30:57.904081 8892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 18:30:57 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127457, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:57.919543 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "84ac0efc7d9645d1a4355aa8bb6ec636"
format_stamp: "Formatted at 2026-08-07 18:30:57 on dist-test-slave-1xrh"
server_key: "a34872be354cd231b4c8c3d0cdc72450"
server_key_iv: "de2b7d28f1951474420710cd036e37ab"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:57.920401 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "84ac0efc7d9645d1a4355aa8bb6ec636"
format_stamp: "Formatted at 2026-08-07 18:30:57 on dist-test-slave-1xrh"
server_key: "a34872be354cd231b4c8c3d0cdc72450"
server_key_iv: "de2b7d28f1951474420710cd036e37ab"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:57.926069 8892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:30:57.939890 8909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:57.941478 8892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:30:57.941679 8892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "84ac0efc7d9645d1a4355aa8bb6ec636"
format_stamp: "Formatted at 2026-08-07 18:30:57 on dist-test-slave-1xrh"
server_key: "a34872be354cd231b4c8c3d0cdc72450"
server_key_iv: "de2b7d28f1951474420710cd036e37ab"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:57.941969 8892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:30:57.961010 8892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:30:57.965126 8892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:30:57.965380 8892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:30:57.966269 8892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:30:57.967620 8892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:30:57.967694 8892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:57.967782 8892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:30:57.967826 8892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:58.002741 8892 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:41253
I20260807 18:30:58.002782 9043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:41253 every 8 connection(s)
I20260807 18:30:58.004166 8892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:30:58.011050 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 8892
I20260807 18:30:58.011266 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:30:58.011894 27655 external_mini_cluster.cc:1428] Setting key 896258941f66f81b9ee2e9fae7ed0e7a
Aug 07 18:30:58 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127457, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:30:58.023980 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:58.006717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:37203 (local address 127.27.1.254:35579)
0807 18:30:58.007292 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:58.007297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:58.008113 (+ 816us) server_negotiation.cc:407] Connection header received
0807 18:30:58.009864 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:58.009869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:58.009963 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:58.010086 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:58.011986 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:58.012876 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:58.015029 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:58.015547 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:58.015681 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:58.018845 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:58.018850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:58.018853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:58.018884 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:58.021432 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:58.022082 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:58.022085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:58.022088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:58.022203 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:58.022565 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:58.022569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:58.022570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:58.022815 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:58.022923 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:58.023553 (+ 630us) server_negotiation.cc:299] Negotiation successful
0807 18:30:58.023708 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":256,"threads_started":1}
I20260807 18:30:58.024883 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:58.007202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35579 (local address 127.27.1.195:37203)
0807 18:30:58.007950 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:58.007987 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:58.009594 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:58.010315 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:58.010324 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:58.010906 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:30:58.011772 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:58.011797 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:58.013028 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:58.013033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:58.014879 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:58.014891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:58.015765 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:58.015772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:58.016019 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:58.016603 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:30:58.016625 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:30:58.018635 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:30:58.021553 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:58.021560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:58.021571 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:58.021986 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:58.022295 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:58.022298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:58.022300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:58.022463 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:58.022919 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:30:58.022924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:30:58.023247 (+ 323us) client_negotiation.cc:769] Sending connection context
0807 18:30:58.023478 (+ 231us) client_negotiation.cc:240] Negotiation successful
0807 18:30:58.023837 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":310,"threads_started":1}
I20260807 18:30:58.026261 9044 heartbeater.cc:347] Connected to a master server at 127.27.1.254:35579
I20260807 18:30:58.026568 9044 heartbeater.cc:464] Registering TS with master...
I20260807 18:30:58.027400 9044 heartbeater.cc:511] Master 127.27.1.254:35579 requested a full tablet report, sending...
I20260807 18:30:58.028767 8493 ts_manager.cc:194] Registered new tserver with Master: 84ac0efc7d9645d1a4355aa8bb6ec636 (127.27.1.195:41253)
I20260807 18:30:58.029659 8493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:37203
I20260807 18:30:58.039490 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:30:58.041795 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 8580
Aug 07 18:30:58 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:30:58.057924 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:58.042647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.27.1.193:40565)
0807 18:30:58.043272 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:58.043278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:58.043303 (+ 25us) server_negotiation.cc:407] Connection header received
0807 18:30:58.043434 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:58.043440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:58.043640 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:58.043762 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:58.045136 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:58.046258 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:58.047993 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:58.048538 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:58.048685 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:58.052816 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:58.052828 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:58.052840 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:58.052872 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:58.055542 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:58.056184 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:58.056190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:58.056192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:58.056324 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:58.056659 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:58.056662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:58.056664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:58.056979 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:58.057201 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:58.057448 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 18:30:58.057650 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":287,"threads_started":1}
I20260807 18:30:58.059274 8719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:30:58.142567 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 8580
I20260807 18:30:58.161891 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 8730
Aug 07 18:30:58 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:30:58.184607 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:58.167262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.27.1.194:45219)
0807 18:30:58.167821 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:58.167829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:58.167855 (+ 26us) server_negotiation.cc:407] Connection header received
0807 18:30:58.167920 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:58.167927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:58.168142 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:58.168357 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:58.169847 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:58.170956 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:58.172441 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:58.173066 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:58.173217 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:58.178267 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:58.178285 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:58.178300 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:58.178346 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:58.181587 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:58.182307 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:58.182313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:58.182316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:58.182459 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:58.182871 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:58.182875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:58.182877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:58.183318 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:58.183632 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:58.183963 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 18:30:58.184254 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":265,"threads_started":1}
I20260807 18:30:58.185837 8880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:30:58.268829 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 8730
I20260807 18:30:58.287909 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 8892
Aug 07 18:30:58 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:30:58.304723 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:58.288831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.27.1.195:41253)
0807 18:30:58.289421 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:58.289428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:58.289456 (+ 28us) server_negotiation.cc:407] Connection header received
0807 18:30:58.289518 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:58.289525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:58.289740 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:58.289915 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:58.291372 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:58.292384 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:58.293770 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:58.294336 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:58.294481 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:58.298986 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:58.298998 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:58.299014 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:58.299053 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:58.301775 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:58.302469 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:58.302477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:58.302479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:58.302609 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:58.303010 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:58.303017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:58.303025 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:58.303429 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:58.303772 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:58.304099 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 18:30:58.304369 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":284,"threads_started":1}
I20260807 18:30:58.305979 9039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:30:58.388332 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 8892
I20260807 18:30:58.406787 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 8460
I20260807 18:30:58.408131 8524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:30:58.516598 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 8460
I20260807 18:30:58.542614 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35579
--webserver_interface=127.27.1.254
--webserver_port=39271
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:35579
--ranger_config_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:54915/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.1.212:54915/kms with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:30:58.872850 9058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:30:58.873135 9058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:30:58.873268 9058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:30:58.880510 9058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:30:58.880635 9058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:30:58.880734 9058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:30:58.880834 9058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:30:58.880918 9058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:30:58.891144 9058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.1.212:54915/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:35579
--ranger_config_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=39271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:30:58.892352 9058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:30:58.893687 9058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:30:58.905525 9065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:58.905797 9068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:58.905609 9064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:58.906791 9058 server_base.cc:1034] running on GCE node
I20260807 18:30:58.907621 9058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:30:58.909070 9058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:30:58.910305 9058 hybrid_clock.cc:648] HybridClock initialized: now 1786127458910270 us; error 57 us; skew 500 ppm
I20260807 18:30:58.910660 9058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:30:58 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127458, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:58.913760 9058 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:30:58.915292 9058 webserver.cc:460] Webserver started at http://127.27.1.254:39271/ using document root <none> and password file <none>
I20260807 18:30:58.915927 9058 fs_manager.cc:359] Metadata directory not provided
I20260807 18:30:58.916079 9058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 18:30:58 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127458, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:59.007812 9058 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 18:30:59.011145 9077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:59.012753 9058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 18:30:59.012984 9058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "df5f30b0378d4095b8844766dd816d5f"
format_stamp: "Formatted at 2026-08-07 18:30:54 on dist-test-slave-1xrh"
server_key: "e5c44676bcb046d17573c4b36e8ed4fa"
server_key_iv: "b0ac90021f82e4738f2cf7f074b39cf6"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:59.014051 9058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:30:59.037537 9058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:30:59.041658 9058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:30:59.041951 9058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:30:59.060828 9058 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:35579
I20260807 18:30:59.060850 9134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:35579 every 8 connection(s)
I20260807 18:30:59.062366 9058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:30:59.069084 9135 sys_catalog.cc:263] Verifying existing consensus state
I20260807 18:30:59.069896 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 9058
I20260807 18:30:59.070993 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:54915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:40565
--local_ip_for_outbound_sockets=127.27.1.193
--tserver_master_addrs=127.27.1.254:35579
--webserver_port=33075
--webserver_interface=127.27.1.193
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.1.212:54915/kms with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:30:59.074095 9135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: Bootstrap starting.
I20260807 18:30:59.095113 9135 log.cc:824] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: Log is configured to *not* fsync() on all Append() calls
I20260807 18:30:59.104249 9135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 18:30:59.104799 9135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: Bootstrap complete.
I20260807 18:30:59.113758 9135 raft_consensus.cc:348] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } }
I20260807 18:30:59.114553 9135 raft_consensus.cc:729] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5f30b0378d4095b8844766dd816d5f, State: Initialized, Role: FOLLOWER
I20260807 18:30:59.115254 9135 consensus_queue.cc:260] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } }
I20260807 18:30:59.115487 9135 raft_consensus.cc:388] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:30:59.115623 9135 raft_consensus.cc:482] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:30:59.115816 9135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 1 FOLLOWER]: Advancing to term 2
I20260807 18:30:59.119722 9135 raft_consensus.cc:504] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } }
I20260807 18:30:59.120203 9135 leader_election.cc:302] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df5f30b0378d4095b8844766dd816d5f; no voters:
I20260807 18:30:59.120823 9135 leader_election.cc:288] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 18:30:59.120966 9139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 2 FOLLOWER]: Leader election won for term 2
I20260807 18:30:59.123196 9135 sys_catalog.cc:564] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:30:59.127447 9139 raft_consensus.cc:686] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [term 2 LEADER]: Becoming Leader. State: Replica: df5f30b0378d4095b8844766dd816d5f, State: Running, Role: LEADER
I20260807 18:30:59.128208 9139 consensus_queue.cc:237] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } }
I20260807 18:30:59.131418 9141 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader df5f30b0378d4095b8844766dd816d5f. Latest consensus state: current_term: 2 leader_uuid: "df5f30b0378d4095b8844766dd816d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } } }
I20260807 18:30:59.131951 9141 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [sys.catalog]: This master's current role is: LEADER
I20260807 18:30:59.131666 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "df5f30b0378d4095b8844766dd816d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5f30b0378d4095b8844766dd816d5f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35579 } } }
I20260807 18:30:59.132091 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f [sys.catalog]: This master's current role is: LEADER
I20260807 18:30:59.133419 9148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:30:59.142292 9148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:30:59.143955 9148 catalog_manager.cc:1259] Loaded cluster ID: c9e526a96e2f4951a09f869f5b4105ca
I20260807 18:30:59.144114 9148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:30:59.148972 9148 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:30:59.150039 9135 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 18:30:59.150401 9148 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: Loaded TSK: 0
I20260807 18:30:59.151954 9148 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 18:30:59.485267 9137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:30:59.485723 9137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:30:59.485965 9137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:30:59.497641 9137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:30:59.497843 9137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:30:59.498085 9137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:30:59.513836 9137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.1.212:54915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:40565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=33075
--tserver_master_addrs=127.27.1.254:35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:30:59.515746 9137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:30:59.517468 9137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:30:59.563702 9176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:30:59.566125 9175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:59.568140 9137 server_base.cc:1034] running on GCE node
W20260807 18:30:59.568782 9178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:30:59.569604 9137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:30:59.570802 9137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:30:59.572108 9137 hybrid_clock.cc:648] HybridClock initialized: now 1786127459572033 us; error 56 us; skew 500 ppm
I20260807 18:30:59.572490 9137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:30:59 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127459, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:30:59.576596 9137 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:30:59.586270 9137 webserver.cc:460] Webserver started at http://127.27.1.193:33075/ using document root <none> and password file <none>
I20260807 18:30:59.587426 9137 fs_manager.cc:359] Metadata directory not provided
I20260807 18:30:59.587661 9137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 18:30:59 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127459, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:30:59.614961 9137 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:30:59.619359 9186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:59.620898 9137 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 18:30:59.621104 9137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "db2572262af44aa29dd509c132933c57"
format_stamp: "Formatted at 2026-08-07 18:30:56 on dist-test-slave-1xrh"
server_key: "eef70df10641151d8b7fb97da960bc78"
server_key_iv: "50dee5d5a58e88490e5d4484f0dd18c3"
server_key_version: "kuduclusterkey@0"
I20260807 18:30:59.622174 9137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:30:59.679878 9137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:30:59.685490 9137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:30:59.685890 9137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:30:59.699592 9137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:30:59.705864 9137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:30:59.706013 9137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:59.706148 9137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:30:59.706243 9137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:30:59.835415 9137 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:40565
I20260807 18:30:59.835481 9305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:40565 every 8 connection(s)
I20260807 18:30:59.837291 9137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:30:59.844640 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 9137
I20260807 18:30:59.845479 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:54915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:45219
--local_ip_for_outbound_sockets=127.27.1.194
--tserver_master_addrs=127.27.1.254:35579
--webserver_port=43235
--webserver_interface=127.27.1.194
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.1.212:54915/kms with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:30:59 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127459, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:30:59.872991 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:59.840662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35579 (local address 127.27.1.193:58633)
0807 18:30:59.842015 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0807 18:30:59.842086 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0807 18:30:59.845508 (+ 3422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:30:59.847866 (+ 2358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:30:59.847876 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:30:59.848447 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:30:59.848965 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:59.848983 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:59.851449 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:59.851456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:59.852878 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:30:59.852891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:30:59.857726 (+ 4835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:30:59.857735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:30:59.858045 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:59.858951 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:30:59.858987 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:30:59.861903 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:30:59.867221 (+ 5318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:59.867232 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:59.867250 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:59.867859 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:59.868390 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:30:59.868394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:30:59.868398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:30:59.868640 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:30:59.869808 (+ 1168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:30:59.869820 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:30:59.870321 (+ 501us) client_negotiation.cc:769] Sending connection context
0807 18:30:59.870645 (+ 324us) client_negotiation.cc:240] Negotiation successful
0807 18:30:59.871332 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":401,"threads_started":1}
I20260807 18:30:59.876703 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:30:59.840521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:58633 (local address 127.27.1.254:35579)
0807 18:30:59.843720 (+ 3199us) server_negotiation.cc:206] Beginning negotiation
0807 18:30:59.843738 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 18:30:59.843807 (+ 69us) server_negotiation.cc:407] Connection header received
0807 18:30:59.846269 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:30:59.846287 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:30:59.846757 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:30:59.847167 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:30:59.849169 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:59.851220 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:59.853040 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:30:59.855371 (+ 2331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:30:59.858556 (+ 3185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:30:59.862260 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:30:59.862274 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:30:59.862298 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:30:59.862350 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:30:59.867014 (+ 4664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:59.868058 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:59.868067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:59.868075 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:59.868259 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:30:59.868814 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:30:59.868819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:30:59.868823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:30:59.869546 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:30:59.869748 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 18:30:59.873800 (+ 4052us) server_negotiation.cc:299] Negotiation successful
0807 18:30:59.874312 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2920,"thread_start_us":2670,"threads_started":1}
I20260807 18:30:59.881072 9306 heartbeater.cc:347] Connected to a master server at 127.27.1.254:35579
I20260807 18:30:59.881433 9306 heartbeater.cc:464] Registering TS with master...
I20260807 18:30:59.882333 9306 heartbeater.cc:511] Master 127.27.1.254:35579 requested a full tablet report, sending...
W20260807 18:30:59.886040 9306 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:35579 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 18:30:59.886297 9306 heartbeater.cc:464] Registering TS with master...
I20260807 18:30:59.886613 9306 heartbeater.cc:511] Master 127.27.1.254:35579 requested a full tablet report, sending...
I20260807 18:30:59.888087 9306 heartbeater.cc:464] Registering TS with master...
I20260807 18:30:59.888422 9306 heartbeater.cc:511] Master 127.27.1.254:35579 requested a full tablet report, sending...
W20260807 18:30:59.889598 9306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 18:31:00.116268 9317 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260807 18:31:00.277240 9310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:00.277554 9310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:00.277715 9310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:00.285214 9310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:00.285352 9310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:00.285537 9310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:31:00.296077 9310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.1.212:54915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:45219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=43235
--tserver_master_addrs=127.27.1.254:35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:00.297531 9310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:00.299068 9310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:00.314284 9322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:00.314226 9323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:00.316393 9325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:00.316879 9310 server_base.cc:1034] running on GCE node
I20260807 18:31:00.317509 9310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:00.319873 9310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:00.321115 9310 hybrid_clock.cc:648] HybridClock initialized: now 1786127460321051 us; error 56 us; skew 500 ppm
I20260807 18:31:00.321436 9310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:00 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127460, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:00.325392 9310 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:31:00.328575 9310 webserver.cc:460] Webserver started at http://127.27.1.194:43235/ using document root <none> and password file <none>
I20260807 18:31:00.329491 9310 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:00.329715 9310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 18:31:00 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127460, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:31:00.353194 9310 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.001s
I20260807 18:31:00.356462 9335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:00.358438 9310 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 18:31:00.358659 9310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "75e79053c5894d988ae6a0340818f0f2"
format_stamp: "Formatted at 2026-08-07 18:30:57 on dist-test-slave-1xrh"
server_key: "f27f6e842724a5ba23391eca0ad42eef"
server_key_iv: "5c9aae6cb6c717247c8a531ba59e3084"
server_key_version: "kuduclusterkey@0"
I20260807 18:31:00.360136 9310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:00.384786 9310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:00.390314 9310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:00.390676 9310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:00.392050 9310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:00.393935 9310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:00.394186 9310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 18:31:00.394356 9310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:00.394482 9310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:00.489233 9310 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:45219
I20260807 18:31:00.489318 9458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:45219 every 8 connection(s)
I20260807 18:31:00.491156 9310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:31:00.495759 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 9310
I20260807 18:31:00.496567 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.1.212:54915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:41253
--local_ip_for_outbound_sockets=127.27.1.195
--tserver_master_addrs=127.27.1.254:35579
--webserver_port=39415
--webserver_interface=127.27.1.195
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.1.212:54915/kms with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:31:00 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127460, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
Aug 07 18:31:00 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786127460, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:00.524464 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:00.499308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:50325 (local address 127.27.1.254:35579)
0807 18:31:00.499811 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:00.499819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:00.499847 (+ 28us) server_negotiation.cc:407] Connection header received
0807 18:31:00.502689 (+ 2842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:00.502695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:00.502789 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:00.502916 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:00.505073 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:00.506216 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:00.508762 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:00.509420 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:00.509558 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:00.514247 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:00.514256 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:00.514264 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:00.514326 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:00.518343 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:00.519419 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:00.519428 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:00.519434 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:00.519588 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:00.520123 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:00.520130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:00.520137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:00.520552 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:00.520702 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:00.523732 (+ 3030us) server_negotiation.cc:299] Negotiation successful
0807 18:31:00.523998 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":1165,"threads_started":1}
I20260807 18:31:00.525271 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:00.494766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35579 (local address 127.27.1.194:50325)
0807 18:31:00.499686 (+ 4920us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:00.499750 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:00.502252 (+ 2502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:00.503392 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:00.503405 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:00.504042 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:00.504852 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:00.504890 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:00.506410 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:00.506416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:00.508561 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:00.508576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:00.509852 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:00.509860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:00.510162 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:00.510968 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:00.511002 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:00.513972 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:00.518523 (+ 4551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:00.518534 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:00.518554 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:00.519264 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:00.519722 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:00.519729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:00.519736 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:00.519976 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:00.520798 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:00.520808 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:00.521229 (+ 421us) client_negotiation.cc:769] Sending connection context
0807 18:31:00.521564 (+ 335us) client_negotiation.cc:240] Negotiation successful
0807 18:31:00.522147 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4580,"thread_start_us":4764,"threads_started":1}
I20260807 18:31:00.527386 9459 heartbeater.cc:347] Connected to a master server at 127.27.1.254:35579
I20260807 18:31:00.527721 9459 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:00.528702 9459 heartbeater.cc:511] Master 127.27.1.254:35579 requested a full tablet report, sending...
I20260807 18:31:00.531061 9097 ts_manager.cc:194] Registered new tserver with Master: 75e79053c5894d988ae6a0340818f0f2 (127.27.1.194:45219)
I20260807 18:31:00.533634 9097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:50325
I20260807 18:31:00.889767 9306 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:00.890273 9306 heartbeater.cc:511] Master 127.27.1.254:35579 requested a full tablet report, sending...
I20260807 18:31:00.891820 9097 ts_manager.cc:194] Registered new tserver with Master: db2572262af44aa29dd509c132933c57 (127.27.1.193:40565)
I20260807 18:31:00.893081 9097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:58633
W20260807 18:31:00.961851 9463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:00.962335 9463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:00.962525 9463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:00.972769 9463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:00.972960 9463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:00.973205 9463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:31:00.988590 9463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.1.212:54915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:41253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=39415
--tserver_master_addrs=127.27.1.254:35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:00.990422 9463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:00.992306 9463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 07 18:31:00 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127460, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
W20260807 18:31:01.023044 9471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:01.029817 9463 server_base.cc:1034] running on GCE node
W20260807 18:31:01.030745 9474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:01.031440 9469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:01.032410 9463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:01.033924 9463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:01.035235 9463 hybrid_clock.cc:648] HybridClock initialized: now 1786127461035162 us; error 58 us; skew 500 ppm
I20260807 18:31:01.035660 9463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:01 dist-test-slave-1xrh krb5kdc[7977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127461, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:01.040048 9463 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:31:01.071542 9463 webserver.cc:460] Webserver started at http://127.27.1.195:39415/ using document root <none> and password file <none>
I20260807 18:31:01.072675 9463 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:01.072937 9463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 18:31:01 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127461, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:31:01.094466 9463 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 18:31:01.097479 9483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:01.098965 9463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:31:01.099196 9463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "84ac0efc7d9645d1a4355aa8bb6ec636"
format_stamp: "Formatted at 2026-08-07 18:30:57 on dist-test-slave-1xrh"
server_key: "a34872be354cd231b4c8c3d0cdc72450"
server_key_iv: "de2b7d28f1951474420710cd036e37ab"
server_key_version: "kuduclusterkey@0"
I20260807 18:31:01.100209 9463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:01.121150 9463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:01.126377 9463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:01.126739 9463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:01.134436 9463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:01.142589 9463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:01.142771 9463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:01.142939 9463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:01.143059 9463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 18:31:01 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127460, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for HTTP/127.27.1.212@KRBTEST.COM
I20260807 18:31:01.538733 9459 heartbeater.cc:503] Master 127.27.1.254:35579 was elected leader, sending a full tablet report...
I20260807 18:31:01.619102 9463 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:41253
I20260807 18:31:01.619081 9603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:41253 every 8 connection(s)
I20260807 18:31:01.621241 9463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:31:01.627781 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 9463
Aug 07 18:31:01 dist-test-slave-1xrh krb5kdc[7977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127461, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:01.672645 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.625480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35579 (local address 127.27.1.195:51167)
0807 18:31:01.640984 (+ 15504us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:01.641052 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:01.644812 (+ 3760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:01.646416 (+ 1604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:01.646429 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:01.647157 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:01.648043 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.648079 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.649744 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.649749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.651755 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.651771 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.653231 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.653242 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.653537 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.654817 (+ 1280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:01.654861 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:01.661783 (+ 6922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:01.667604 (+ 5821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:01.667614 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:01.667634 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:01.668286 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:01.668756 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:01.668761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:01.668765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:01.669022 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:01.669769 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:01.669778 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:01.670219 (+ 441us) client_negotiation.cc:769] Sending connection context
0807 18:31:01.670543 (+ 324us) client_negotiation.cc:240] Negotiation successful
0807 18:31:01.671197 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2186,"thread_start_us":1918,"threads_started":1}
I20260807 18:31:01.673794 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.624634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:51167 (local address 127.27.1.254:35579)
0807 18:31:01.643479 (+ 18845us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:01.643486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:01.643509 (+ 23us) server_negotiation.cc:407] Connection header received
0807 18:31:01.645281 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:01.645287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:01.645412 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:01.645526 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:01.648276 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.649514 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.651948 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.652591 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.652697 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.662107 (+ 9410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:01.662114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:01.662119 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:01.662159 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:01.667430 (+ 5271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:01.668459 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:01.668465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:01.668469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:01.668618 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:01.669168 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:01.669174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:01.669177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:01.669539 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:01.669691 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:01.673172 (+ 3481us) server_negotiation.cc:299] Negotiation successful
0807 18:31:01.673408 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18715,"thread_start_us":18578,"threads_started":1}
I20260807 18:31:01.676054 9604 heartbeater.cc:347] Connected to a master server at 127.27.1.254:35579
I20260807 18:31:01.676398 9604 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:01.677290 9604 heartbeater.cc:511] Master 127.27.1.254:35579 requested a full tablet report, sending...
I20260807 18:31:01.679178 9097 ts_manager.cc:194] Registered new tserver with Master: 84ac0efc7d9645d1a4355aa8bb6ec636 (127.27.1.195:41253)
I20260807 18:31:01.680380 9097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:51167
I20260807 18:31:01.684182 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.643439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.27.1.254:35579)
0807 18:31:01.654354 (+ 10915us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:01.654360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:01.654379 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:31:01.654428 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:01.654436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:01.654516 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:01.655742 (+ 1226us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:01.661631 (+ 5889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.662924 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.665026 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.665857 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.666666 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.669691 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:01.669699 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:01.669704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:01.669751 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:01.679730 (+ 9979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:01.680798 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:01.680803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:01.680806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:01.680975 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:01.681480 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:01.681485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:01.681488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:01.681817 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:01.683329 (+ 1512us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:01.683531 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 18:31:01.683764 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10781,"thread_start_us":10635,"threads_started":1}
I20260807 18:31:01.686925 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:31:01.726146 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.701911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.27.1.254:35579)
0807 18:31:01.702132 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:01.702139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:01.706029 (+ 3890us) server_negotiation.cc:407] Connection header received
0807 18:31:01.706364 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:01.706370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:01.706456 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:01.706583 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:01.708928 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.710057 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.712663 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.713318 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.713443 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.717645 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:01.717652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:01.717658 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:01.717700 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:01.722441 (+ 4741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:01.723526 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:01.723532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:01.723536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:01.723715 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:01.724275 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:01.724281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:01.724285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:01.724658 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:01.724913 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:01.725557 (+ 644us) server_negotiation.cc:299] Negotiation successful
0807 18:31:01.725809 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:31:01.733764 9097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:31:01.745268 9097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:31:01.809055 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.794953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40565 (local address 127.0.0.1:36698)
0807 18:31:01.797512 (+ 2559us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:01.797532 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:01.797665 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:01.799285 (+ 1620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:01.799291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:01.799317 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:01.800919 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.800938 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.803004 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.803009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.805162 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.805178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.807474 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.807480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.808225 (+ 745us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.808283 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 18:31:01.808424 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 18:31:01.808626 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2428,"mutex_wait_us":1,"thread_start_us":8207,"threads_started":1}
I20260807 18:31:01.809254 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.763624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.27.1.195:41253)
0807 18:31:01.779702 (+ 16078us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:01.779712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:01.779748 (+ 36us) server_negotiation.cc:407] Connection header received
0807 18:31:01.779848 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:01.779857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:01.780131 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:01.780355 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:01.796590 (+ 16235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.798279 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.803520 (+ 5241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.805197 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.805313 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.805578 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:01.808697 (+ 3119us) server_negotiation.cc:299] Negotiation successful
0807 18:31:01.808884 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15864,"thread_start_us":16994,"threads_started":1}
I20260807 18:31:01.810202 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.789646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.27.1.193:40565)
0807 18:31:01.795700 (+ 6054us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:01.795710 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:01.797796 (+ 2086us) server_negotiation.cc:407] Connection header received
0807 18:31:01.797906 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:01.797915 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:01.798194 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:01.798422 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:01.801194 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.802806 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.805369 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.806907 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.807038 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.807275 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:01.809614 (+ 2339us) server_negotiation.cc:299] Negotiation successful
0807 18:31:01.809809 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5845,"thread_start_us":5677,"threads_started":1}
I20260807 18:31:01.810096 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.762955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:41253 (local address 127.0.0.1:44582)
0807 18:31:01.778451 (+ 15496us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:01.778512 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:01.778709 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:01.795839 (+ 17130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:01.795847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:01.795886 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:01.796320 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.796337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.798491 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.798497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.802956 (+ 4459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.802972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.805909 (+ 2937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.805915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.806106 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.808502 (+ 2396us) client_negotiation.cc:769] Sending connection context
0807 18:31:01.809539 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0807 18:31:01.809687 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15150,"mutex_wait_us":2321,"thread_start_us":15971,"threads_started":1}
I20260807 18:31:01.810778 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.786083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:45219 (local address 127.0.0.1:59148)
0807 18:31:01.789158 (+ 3075us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:01.789180 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:01.789301 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:01.800726 (+ 11425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:01.800731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:01.800757 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:01.801261 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.801274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.806106 (+ 4832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.806112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.807965 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.807976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.809937 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.809943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.810146 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.810184 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 18:31:01.810258 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 18:31:01.810394 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2931,"mutex_wait_us":21,"thread_start_us":2776,"threads_started":1}
I20260807 18:31:01.811947 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.780186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.27.1.194:45219)
0807 18:31:01.800157 (+ 19971us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:01.800166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:01.800194 (+ 28us) server_negotiation.cc:407] Connection header received
0807 18:31:01.800293 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:01.800301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:01.800525 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:01.800754 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:01.804393 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.805938 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.808122 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.809664 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.811149 (+ 1485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.811380 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:01.811464 (+ 84us) server_negotiation.cc:299] Negotiation successful
0807 18:31:01.811623 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19780,"thread_start_us":19703,"threads_started":1}
I20260807 18:31:01.815050 9533 tablet_service.cc:1467] Processing CreateTablet for tablet 9a197cc8a6544a0d950a3a186ee4df8a (DEFAULT_TABLE table=test-table [id=f51c9463a3ae44be999fb21e7f19cb96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:01.815598 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 9a197cc8a6544a0d950a3a186ee4df8a (DEFAULT_TABLE table=test-table [id=f51c9463a3ae44be999fb21e7f19cb96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:01.816195 9389 tablet_service.cc:1467] Processing CreateTablet for tablet 9a197cc8a6544a0d950a3a186ee4df8a (DEFAULT_TABLE table=test-table [id=f51c9463a3ae44be999fb21e7f19cb96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:01.816911 9533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a197cc8a6544a0d950a3a186ee4df8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:01.817052 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a197cc8a6544a0d950a3a186ee4df8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:01.817729 9389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a197cc8a6544a0d950a3a186ee4df8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:01.837819 9635 tablet_bootstrap.cc:492] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: Bootstrap starting.
I20260807 18:31:01.839244 9632 tablet_bootstrap.cc:492] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57: Bootstrap starting.
I20260807 18:31:01.841703 9633 tablet_bootstrap.cc:492] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: Bootstrap starting.
I20260807 18:31:01.844031 9635 tablet_bootstrap.cc:654] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:01.845424 9633 tablet_bootstrap.cc:654] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:01.845393 9635 log.cc:824] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:01.846832 9633 log.cc:824] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:01.850081 9635 tablet_bootstrap.cc:492] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: No bootstrap required, opened a new log
I20260807 18:31:01.850230 9632 tablet_bootstrap.cc:654] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:01.850383 9635 ts_tablet_manager.cc:1397] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260807 18:31:01.851619 9632 log.cc:824] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:01.857841 9632 tablet_bootstrap.cc:492] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57: No bootstrap required, opened a new log
I20260807 18:31:01.858197 9632 ts_tablet_manager.cc:1397] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20260807 18:31:01.861829 9635 raft_consensus.cc:348] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 } } peers { permanent_uuid: "75e79053c5894d988ae6a0340818f0f2" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 45219 } } peers { permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 } }
I20260807 18:31:01.862309 9635 raft_consensus.cc:374] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:01.862422 9635 raft_consensus.cc:729] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e79053c5894d988ae6a0340818f0f2, State: Initialized, Role: FOLLOWER
I20260807 18:31:01.863083 9635 consensus_queue.cc:260] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 } } peers { permanent_uuid: "75e79053c5894d988ae6a0340818f0f2" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 45219 } } peers { permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 } }
I20260807 18:31:01.865965 9633 tablet_bootstrap.cc:492] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: No bootstrap required, opened a new log
I20260807 18:31:01.866288 9633 ts_tablet_manager.cc:1397] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.004s
I20260807 18:31:01.874740 9632 raft_consensus.cc:348] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 } } peers { permanent_uuid: "75e79053c5894d988ae6a0340818f0f2" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 45219 } } peers { permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 } }
I20260807 18:31:01.876313 9633 raft_consensus.cc:348] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 } } peers { permanent_uuid: "75e79053c5894d988ae6a0340818f0f2" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 45219 } } peers { permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 } }
I20260807 18:31:01.876744 9635 ts_tablet_manager.cc:1428] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: Time spent starting tablet: real 0.026s user 0.022s sys 0.003s
I20260807 18:31:01.876756 9633 raft_consensus.cc:374] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:01.876884 9633 raft_consensus.cc:729] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ac0efc7d9645d1a4355aa8bb6ec636, State: Initialized, Role: FOLLOWER
I20260807 18:31:01.877292 9632 raft_consensus.cc:374] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:01.877480 9632 raft_consensus.cc:729] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2572262af44aa29dd509c132933c57, State: Initialized, Role: FOLLOWER
I20260807 18:31:01.877586 9633 consensus_queue.cc:260] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 } } peers { permanent_uuid: "75e79053c5894d988ae6a0340818f0f2" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 45219 } } peers { permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 } }
I20260807 18:31:01.878676 9604 heartbeater.cc:503] Master 127.27.1.254:35579 was elected leader, sending a full tablet report...
I20260807 18:31:01.879179 9633 ts_tablet_manager.cc:1428] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260807 18:31:01.880571 9632 consensus_queue.cc:260] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 } } peers { permanent_uuid: "75e79053c5894d988ae6a0340818f0f2" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 45219 } } peers { permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 } }
I20260807 18:31:01.881729 9306 heartbeater.cc:503] Master 127.27.1.254:35579 was elected leader, sending a full tablet report...
I20260807 18:31:01.882269 9632 ts_tablet_manager.cc:1428] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57: Time spent starting tablet: real 0.024s user 0.016s sys 0.000s
W20260807 18:31:01.894713 9605 tablet.cc:2369] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:01.906559 9640 raft_consensus.cc:482] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:31:01.906845 9640 raft_consensus.cc:504] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 } } peers { permanent_uuid: "75e79053c5894d988ae6a0340818f0f2" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 45219 } } peers { permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 } }
I20260807 18:31:01.908586 9640 leader_election.cc:288] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84ac0efc7d9645d1a4355aa8bb6ec636 (127.27.1.195:41253), db2572262af44aa29dd509c132933c57 (127.27.1.193:40565)
I20260807 18:31:01.918412 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.908989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:37043 (local address 127.27.1.195:41253)
0807 18:31:01.909202 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:01.909210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:01.909697 (+ 487us) server_negotiation.cc:407] Connection header received
0807 18:31:01.910031 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:01.910039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:01.910134 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:01.910206 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:01.911263 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.913117 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.915999 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.917143 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.917269 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.917340 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:01.918018 (+ 678us) server_negotiation.cc:299] Negotiation successful
0807 18:31:01.918154 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 18:31:01.919180 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.908937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:41253 (local address 127.27.1.194:37043)
0807 18:31:01.909538 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:01.909554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:01.909833 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:01.910455 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:01.910459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:01.910489 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:01.911053 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.911071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.913314 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.913320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.915805 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.915820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.917540 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.917547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.917802 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.917837 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 18:31:01.918600 (+ 763us) client_negotiation.cc:240] Negotiation successful
0807 18:31:01.918803 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":313,"threads_started":1}
I20260807 18:31:01.921197 9556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a197cc8a6544a0d950a3a186ee4df8a" candidate_uuid: "75e79053c5894d988ae6a0340818f0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" is_pre_election: true
I20260807 18:31:01.921768 9556 raft_consensus.cc:2454] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e79053c5894d988ae6a0340818f0f2 in term 0.
I20260807 18:31:01.922511 9336 leader_election.cc:302] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e79053c5894d988ae6a0340818f0f2, 84ac0efc7d9645d1a4355aa8bb6ec636; no voters:
I20260807 18:31:01.923100 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.910196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:52809 (local address 127.27.1.193:40565)
0807 18:31:01.910382 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:01.910388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:01.912820 (+ 2432us) server_negotiation.cc:407] Connection header received
0807 18:31:01.913137 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:01.913145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:01.913238 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:01.913309 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:01.915086 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.916347 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.919692 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.921141 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.921274 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.921341 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:01.922632 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0807 18:31:01.922797 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 18:31:01.923804 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.909727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40565 (local address 127.27.1.194:52809)
0807 18:31:01.912701 (+ 2974us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:01.912723 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:01.912933 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:01.913566 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:01.913573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:01.913601 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:01.914884 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.914900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.916572 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.916577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.919513 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:01.919531 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.922162 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.922168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:01.922395 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.922439 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 18:31:01.923151 (+ 712us) client_negotiation.cc:240] Negotiation successful
0807 18:31:01.923448 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2854,"thread_start_us":2664,"threads_started":1}
I20260807 18:31:01.923804 9640 raft_consensus.cc:2781] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:31:01.924033 9640 raft_consensus.cc:482] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:31:01.924160 9640 raft_consensus.cc:3037] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:01.924960 9258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a197cc8a6544a0d950a3a186ee4df8a" candidate_uuid: "75e79053c5894d988ae6a0340818f0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2572262af44aa29dd509c132933c57" is_pre_election: true
I20260807 18:31:01.925532 9258 raft_consensus.cc:2454] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e79053c5894d988ae6a0340818f0f2 in term 0.
I20260807 18:31:01.926764 9640 raft_consensus.cc:504] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 } } peers { permanent_uuid: "75e79053c5894d988ae6a0340818f0f2" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 45219 } } peers { permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 } }
I20260807 18:31:01.928741 9640 leader_election.cc:288] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [CANDIDATE]: Term 1 election: Requested vote from peers 84ac0efc7d9645d1a4355aa8bb6ec636 (127.27.1.195:41253), db2572262af44aa29dd509c132933c57 (127.27.1.193:40565)
I20260807 18:31:01.929221 9556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a197cc8a6544a0d950a3a186ee4df8a" candidate_uuid: "75e79053c5894d988ae6a0340818f0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636"
I20260807 18:31:01.929425 9556 raft_consensus.cc:3037] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:01.929701 9258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a197cc8a6544a0d950a3a186ee4df8a" candidate_uuid: "75e79053c5894d988ae6a0340818f0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2572262af44aa29dd509c132933c57"
I20260807 18:31:01.930085 9258 raft_consensus.cc:3037] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:01.932621 9556 raft_consensus.cc:2454] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e79053c5894d988ae6a0340818f0f2 in term 1.
I20260807 18:31:01.933056 9336 leader_election.cc:302] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e79053c5894d988ae6a0340818f0f2, 84ac0efc7d9645d1a4355aa8bb6ec636; no voters:
I20260807 18:31:01.933337 9640 raft_consensus.cc:2781] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:31:01.934351 9258 raft_consensus.cc:2454] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e79053c5894d988ae6a0340818f0f2 in term 1.
I20260807 18:31:01.942169 9640 raft_consensus.cc:686] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 1 LEADER]: Becoming Leader. State: Replica: 75e79053c5894d988ae6a0340818f0f2, State: Running, Role: LEADER
I20260807 18:31:01.942720 9640 consensus_queue.cc:237] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 } } peers { permanent_uuid: "75e79053c5894d988ae6a0340818f0f2" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 45219 } } peers { permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 } }
I20260807 18:31:01.948618 9096 catalog_manager.cc:5495] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75e79053c5894d988ae6a0340818f0f2 (127.27.1.194). New cstate: current_term: 1 leader_uuid: "75e79053c5894d988ae6a0340818f0f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75e79053c5894d988ae6a0340818f0f2" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 45219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 } health_report { overall_health: UNKNOWN } } }
I20260807 18:31:01.978873 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:01.969158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.27.1.194:45219)
0807 18:31:01.971323 (+ 2165us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:01.971329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:01.971350 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:31:01.971454 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:01.971458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:01.971569 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:01.971719 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:31:01.972477 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.973704 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.976014 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:01.976649 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:01.976793 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:01.977527 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:31:01.978199 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:31:01.978345 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:01.978489 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 18:31:01.978576 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2012}
I20260807 18:31:01.987473 9556 raft_consensus.cc:1261] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636 [term 1 FOLLOWER]: Refusing update from remote peer 75e79053c5894d988ae6a0340818f0f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:31:01.987483 9258 raft_consensus.cc:1261] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57 [term 1 FOLLOWER]: Refusing update from remote peer 75e79053c5894d988ae6a0340818f0f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:31:01.988554 9640 consensus_queue.cc:1035] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db2572262af44aa29dd509c132933c57" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:31:01.988948 9646 consensus_queue.cc:1035] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84ac0efc7d9645d1a4355aa8bb6ec636" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 18:31:02.000487 9460 tablet.cc:2369] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:02.002283 9649 mvcc.cc:205] Tried to move back new op lower bound from 7315978084289998848 to 7315978084122824704. Current Snapshot: MvccSnapshot[applied={T|T < 7315978084289998848}]
I20260807 18:31:02.003333 9650 mvcc.cc:205] Tried to move back new op lower bound from 7315978084289998848 to 7315978084122824704. Current Snapshot: MvccSnapshot[applied={T|T < 7315978084289998848}]
I20260807 18:31:02.020684 9095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:31:02.021030 9095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:31:02.025463 9095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df5f30b0378d4095b8844766dd816d5f: Sending DeleteTablet for 3 replicas of tablet 9a197cc8a6544a0d950a3a186ee4df8a
I20260807 18:31:02.027348 9533 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a197cc8a6544a0d950a3a186ee4df8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:02 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:44582
I20260807 18:31:02.027575 9389 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a197cc8a6544a0d950a3a186ee4df8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:02 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:59148
I20260807 18:31:02.028488 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a197cc8a6544a0d950a3a186ee4df8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:02 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:36698
I20260807 18:31:02.031586 9660 tablet_replica.cc:330] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: stopping tablet replica
I20260807 18:31:02.031963 9660 raft_consensus.cc:2229] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:31:02.032617 9660 raft_consensus.cc:2258] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:02.034978 9659 tablet_replica.cc:330] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: stopping tablet replica
I20260807 18:31:02.035514 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 9137
I20260807 18:31:02.036546 9659 raft_consensus.cc:2229] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:31:02.037729 9660 ts_tablet_manager.cc:1905] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:31:02.038304 9659 raft_consensus.cc:2258] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:02.046191 9660 ts_tablet_manager.cc:1918] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:31:02.047065 9661 tablet_replica.cc:330] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57: stopping tablet replica
I20260807 18:31:02.048609 9659 ts_tablet_manager.cc:1905] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:31:02.049137 9661 raft_consensus.cc:2229] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:31:02.051797 9661 raft_consensus.cc:2258] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:02.057299 9659 ts_tablet_manager.cc:1918] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:31:02.057412 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.037421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.27.1.193:40565)
0807 18:31:02.037640 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:02.037646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:02.037669 (+ 23us) server_negotiation.cc:407] Connection header received
0807 18:31:02.037736 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:02.037742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:02.037852 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:02.037973 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:02.041804 (+ 3831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:02.042883 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:02.044880 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:02.045505 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:02.045628 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:02.049509 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:02.049524 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:02.049542 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:02.049583 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:02.053806 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:02.054722 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:02.054733 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:02.054737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:02.054898 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:02.055404 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:02.055408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:02.055411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:02.055891 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:02.056196 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:02.056841 (+ 645us) server_negotiation.cc:299] Negotiation successful
0807 18:31:02.057063 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 18:31:02.058786 9300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:02.059826 9660 log.cc:1196] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/9a197cc8a6544a0d950a3a186ee4df8a
I20260807 18:31:02.060516 9660 ts_tablet_manager.cc:1939] T 9a197cc8a6544a0d950a3a186ee4df8a P 75e79053c5894d988ae6a0340818f0f2: Deleting consensus metadata
I20260807 18:31:02.062536 9078 catalog_manager.cc:4841] TS 75e79053c5894d988ae6a0340818f0f2 (127.27.1.194:45219): tablet 9a197cc8a6544a0d950a3a186ee4df8a (table test-table [id=f51c9463a3ae44be999fb21e7f19cb96]) successfully deleted
I20260807 18:31:02.062987 9659 log.cc:1196] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/9a197cc8a6544a0d950a3a186ee4df8a
I20260807 18:31:02.063644 9659 ts_tablet_manager.cc:1939] T 9a197cc8a6544a0d950a3a186ee4df8a P 84ac0efc7d9645d1a4355aa8bb6ec636: Deleting consensus metadata
I20260807 18:31:02.066756 9078 catalog_manager.cc:4841] TS 84ac0efc7d9645d1a4355aa8bb6ec636 (127.27.1.195:41253): tablet 9a197cc8a6544a0d950a3a186ee4df8a (table test-table [id=f51c9463a3ae44be999fb21e7f19cb96]) successfully deleted
I20260807 18:31:02.059621 9661 ts_tablet_manager.cc:1905] T 9a197cc8a6544a0d950a3a186ee4df8a P db2572262af44aa29dd509c132933c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:31:02.184719 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 9137
I20260807 18:31:02.209725 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 9310
W20260807 18:31:02.210177 9078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.1.193:40565 (error 108)
I20260807 18:31:02.213209 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.212618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40565 (local address 127.0.0.1:36730)
0807 18:31:02.212888 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:02.213013 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260807 18:31:02.213652 9078 catalog_manager.cc:4584] TS db2572262af44aa29dd509c132933c57 (127.27.1.193:40565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a197cc8a6544a0d950a3a186ee4df8a: Network error: Client connection negotiation failed: client connection to 127.27.1.193:40565: connect: Connection refused (error 111)
I20260807 18:31:02.233886 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.211950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.27.1.194:45219)
0807 18:31:02.212145 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:02.212158 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:02.212181 (+ 23us) server_negotiation.cc:407] Connection header received
0807 18:31:02.212255 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:02.212264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:02.212355 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:02.212485 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:02.214525 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:02.215620 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:02.217213 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:02.217788 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:02.218023 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:02.227408 (+ 9385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:02.227422 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:02.227437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:02.227480 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:02.230580 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:02.231426 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:02.231443 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:02.231445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:02.231595 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:02.232001 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:02.232013 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:02.232015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:02.232493 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:02.232804 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:02.233110 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 18:31:02.233396 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 18:31:02.234689 9454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:02.267822 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.267267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40565 (local address 127.0.0.1:36734)
0807 18:31:02.267506 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:02.267626 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 18:31:02.268951 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.268552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40565 (local address 127.0.0.1:36742)
0807 18:31:02.268710 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:02.268812 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 18:31:02.329661 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.329196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40565 (local address 127.0.0.1:36758)
0807 18:31:02.329391 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:02.329487 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 18:31:02.330714 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.330390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40565 (local address 127.0.0.1:36770)
0807 18:31:02.330534 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:02.330601 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 18:31:02.350464 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 9310
I20260807 18:31:02.373095 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 9463
I20260807 18:31:02.386857 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.374062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.27.1.195:41253)
0807 18:31:02.374280 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:02.374289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:02.374317 (+ 28us) server_negotiation.cc:407] Connection header received
0807 18:31:02.374372 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:02.374375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:02.374458 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:02.374568 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:02.375951 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:02.376693 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:02.378122 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:02.378539 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:02.378676 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:02.381375 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:02.381390 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:02.381406 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:02.381442 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:02.384080 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:02.384764 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:02.384771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:02.384774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:02.384912 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:02.385291 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:02.385296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:02.385300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:02.385676 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:02.386017 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:02.386287 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 18:31:02.386507 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:31:02.387604 9599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:02.411906 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.411342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40565 (local address 127.0.0.1:36772)
0807 18:31:02.411591 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:02.411703 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 18:31:02.412993 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.412572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40565 (local address 127.0.0.1:36774)
0807 18:31:02.412774 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:02.412857 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 18:31:02.478610 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 9463
I20260807 18:31:02.498179 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 9058
I20260807 18:31:02.499390 9130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:02.613135 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:02.612223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40565 (local address 127.0.0.1:36790)
0807 18:31:02.612700 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:02.612839 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313}
I20260807 18:31:02.613225 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 9058
I20260807 18:31:02.639160 27655 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 18:31:02.984320 27655 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 18:31:02.984578 27655 mini_ranger.cc:66] Stopping Ranger...
I20260807 18:31:03.330667 27655 mini_ranger.cc:68] Stopped Ranger
2026-08-07 18:31:03.330 UTC [8008] LOG: received smart shutdown request
2026-08-07 18:31:03.334 UTC [8008] LOG: background worker "logical replication launcher" (PID 8018) exited with exit code 1
2026-08-07 18:31:03.334 UTC [8012] LOG: shutting down
2026-08-07 18:31:03.502 UTC [8008] LOG: database system is shut down
2026-08-07T18:31:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54101 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:31:03 dist-test-slave-1xrh krb5kdc[9677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:31:03 dist-test-slave-1xrh krb5kdc[9677](info): set up 2 sockets
Aug 07 18:31:03 dist-test-slave-1xrh krb5kdc[9677](info): commencing operation
krb5kdc: starting...
W20260807 18:31:04.837339 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:31:05 dist-test-slave-1xrh krb5kdc[9677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:31:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:31:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:31:05.115003 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35933
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:35933,127.27.1.253:41857,127.27.1.252:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:05.449297 9692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:05.449600 9692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:05.449775 9692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:05.458482 9692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:31:05.458585 9692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:05.458688 9692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:05.458791 9692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:31:05.458853 9692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:31:05.468791 9692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:35933,127.27.1.253:41857,127.27.1.252:33999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:05.470145 9692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:05.471586 9692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:05.483369 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:05.483426 9699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:05.483327 9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:05.488473 9692 server_base.cc:1034] running on GCE node
I20260807 18:31:05.489197 9692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:05.490514 9692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:05.491710 9692 hybrid_clock.cc:648] HybridClock initialized: now 1786127465491671 us; error 27 us; skew 500 ppm
I20260807 18:31:05.492061 9692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:05 dist-test-slave-1xrh krb5kdc[9677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:05.495085 9692 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:31:05.496667 9692 webserver.cc:460] Webserver started at http://127.27.1.254:35569/ using document root <none> and password file <none>
I20260807 18:31:05.497362 9692 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:05.497519 9692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:05.497826 9692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:05.500828 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "12f02ed4351749d8aaeec8a320526a73"
format_stamp: "Formatted at 2026-08-07 18:31:05 on dist-test-slave-1xrh"
server_key: "f91a3fcc8b3897f0a8571cf99f095950"
server_key_iv: "a1e6219283af2d83134c024a19362060"
server_key_version: "encryptionkey@0"
I20260807 18:31:05.501641 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "12f02ed4351749d8aaeec8a320526a73"
format_stamp: "Formatted at 2026-08-07 18:31:05 on dist-test-slave-1xrh"
server_key: "f91a3fcc8b3897f0a8571cf99f095950"
server_key_iv: "a1e6219283af2d83134c024a19362060"
server_key_version: "encryptionkey@0"
I20260807 18:31:05.506560 9692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 18:31:05.509902 9709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:05.511205 9692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:31:05.511377 9692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "12f02ed4351749d8aaeec8a320526a73"
format_stamp: "Formatted at 2026-08-07 18:31:05 on dist-test-slave-1xrh"
server_key: "f91a3fcc8b3897f0a8571cf99f095950"
server_key_iv: "a1e6219283af2d83134c024a19362060"
server_key_version: "encryptionkey@0"
I20260807 18:31:05.511579 9692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:05.538877 9692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:05.542855 9692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:05.543192 9692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:05.562091 9692 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:35933
I20260807 18:31:05.562139 9762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:35933 every 8 connection(s)
I20260807 18:31:05.563501 9692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:31:05.567090 9763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:05.570334 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 9692
I20260807 18:31:05.570578 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:31:05.571491 27655 external_mini_cluster.cc:1428] Setting key d33015e6a112bdda827d36d3b523737a
I20260807 18:31:05.575809 9763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:31:05 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.1.253@KRBTEST.COM; defaulting to no policy
I20260807 18:31:05.601570 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.579875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.27.1.254:35933)
0807 18:31:05.580847 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:05.580863 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:05.581412 (+ 549us) server_negotiation.cc:407] Connection header received
0807 18:31:05.583425 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:05.583434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:05.583746 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:05.584008 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:05.585578 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:05.586766 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:05.589067 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:05.589616 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:05.589751 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:05.593711 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:05.593719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:05.593730 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:05.593758 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:05.597462 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:05.598156 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:05.598166 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:05.598170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:05.598368 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:05.598775 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:05.598780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:05.598785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:05.599527 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:05.599758 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:05.600271 (+ 513us) server_negotiation.cc:299] Negotiation successful
0807 18:31:05.600625 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":712,"thread_start_us":459,"threads_started":1}
I20260807 18:31:05.601608 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.579875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35933 (local address 127.0.0.1:54056)
0807 18:31:05.581238 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.581280 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:05.583012 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:05.584033 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:05.584037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:05.584879 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:05.585371 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:05.585397 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:05.586945 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:05.586953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:05.588918 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:05.588946 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:05.589864 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:05.589874 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:05.590129 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:05.590843 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:05.590875 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:05.593487 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:05.597599 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:05.597609 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:05.597618 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:05.598000 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:05.598486 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:05.598489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:05.598491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:05.598648 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:05.599666 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:05.599676 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:05.599857 (+ 181us) client_negotiation.cc:769] Sending connection context
0807 18:31:05.600053 (+ 196us) client_negotiation.cc:240] Negotiation successful
0807 18:31:05.600491 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"spinlock_wait_cycles":75520,"thread_start_us":862,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.253@KRBTEST.COM" created.
I20260807 18:31:05.604482 9763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
I20260807 18:31:05.605479 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.605122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37082)
0807 18:31:05.605276 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.605373 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260807 18:31:05.605702 9713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
I20260807 18:31:05.607287 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.606889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37090)
0807 18:31:05.607061 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.607163 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 18:31:05.607482 9763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:41857: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab.
Entry for principal kudu/127.27.1.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab.
I20260807 18:31:05.635600 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.253:41857
--webserver_interface=127.27.1.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab
--principal=kudu/127.27.1.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:35933,127.27.1.253:41857,127.27.1.252:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:31:05.656850 9763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } attempt: 1
I20260807 18:31:05.658026 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.657560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37098)
0807 18:31:05.657764 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.657879 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 18:31:05.659221 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.658785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37104)
0807 18:31:05.658933 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.659009 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 18:31:05.659411 9763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:41857: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
I20260807 18:31:05.727677 9763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } attempt: 2
I20260807 18:31:05.728847 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.728380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37108)
0807 18:31:05.728595 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.728695 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260807 18:31:05.729956 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.729593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37110)
0807 18:31:05.729743 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.729831 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 18:31:05.730149 9763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:41857: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
I20260807 18:31:05.821431 9763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } attempt: 3
I20260807 18:31:05.822603 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.822148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37126)
0807 18:31:05.822356 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.822465 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 18:31:05.823715 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.823348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37138)
0807 18:31:05.823506 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.823587 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 18:31:05.823922 9763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:41857: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
I20260807 18:31:05.967275 9763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } attempt: 4
I20260807 18:31:05.968485 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.967967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37150)
0807 18:31:05.968270 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.968354 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260807 18:31:05.969532 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:05.969174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37154)
0807 18:31:05.969326 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:05.969411 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 18:31:05.969765 9763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:41857: Network error: Client connection negotiation failed: client connection to 127.27.1.253:41857: connect: Connection refused (error 111)
W20260807 18:31:05.970788 9770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:05.971089 9770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:05.971300 9770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:05.978425 9770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:31:05.978518 9770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:05.978565 9770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:05.978658 9770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:31:05.978745 9770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:31:05.988692 9770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:35933,127.27.1.253:41857,127.27.1.252:33999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab
--principal=kudu/127.27.1.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.253:41857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.1.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:05.990033 9770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:05.991452 9770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:06.004381 9775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:06.004333 9776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:06.004403 9778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:06.005223 9770 server_base.cc:1034] running on GCE node
I20260807 18:31:06.006034 9770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:06.007534 9770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:06.008762 9770 hybrid_clock.cc:648] HybridClock initialized: now 1786127466008725 us; error 47 us; skew 500 ppm
I20260807 18:31:06.009115 9770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:06.012535 9770 init.cc:377] Logged in from keytab as kudu/127.27.1.253@KRBTEST.COM (short username kudu)
I20260807 18:31:06.014083 9770 webserver.cc:460] Webserver started at http://127.27.1.253:44163/ using document root <none> and password file <none>
I20260807 18:31:06.014868 9770 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:06.015028 9770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:06.015378 9770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:06.018471 9770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/data/instance:
uuid: "a6e2941c3cbd423faeec4af3c76b9913"
format_stamp: "Formatted at 2026-08-07 18:31:06 on dist-test-slave-1xrh"
server_key: "599db6ee8481e7c6902901410ef42cac"
server_key_iv: "679abe4c873065f0beb168c59465e647"
server_key_version: "encryptionkey@0"
I20260807 18:31:06.019510 9770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/wal/instance:
uuid: "a6e2941c3cbd423faeec4af3c76b9913"
format_stamp: "Formatted at 2026-08-07 18:31:06 on dist-test-slave-1xrh"
server_key: "599db6ee8481e7c6902901410ef42cac"
server_key_iv: "679abe4c873065f0beb168c59465e647"
server_key_version: "encryptionkey@0"
I20260807 18:31:06.024816 9770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:31:06.028201 9785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:06.029456 9770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:31:06.029644 9770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/wal
uuid: "a6e2941c3cbd423faeec4af3c76b9913"
format_stamp: "Formatted at 2026-08-07 18:31:06 on dist-test-slave-1xrh"
server_key: "599db6ee8481e7c6902901410ef42cac"
server_key_iv: "679abe4c873065f0beb168c59465e647"
server_key_version: "encryptionkey@0"
I20260807 18:31:06.029842 9770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:06.042937 9770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:06.047271 9770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:06.047545 9770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:06.065666 9770 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.253:41857
I20260807 18:31:06.065701 9844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.253:41857 every 8 connection(s)
I20260807 18:31:06.067301 9770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
I20260807 18:31:06.071224 9845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:06.071599 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 9770
I20260807 18:31:06.071790 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-1/wal/instance
I20260807 18:31:06.072273 27655 external_mini_cluster.cc:1428] Setting key 73b79cc4aeabcdecba032b6b24de0686
I20260807 18:31:06.077593 9845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.1.252@KRBTEST.COM; defaulting to no policy
I20260807 18:31:06.099934 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.080223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.27.1.254:35933)
0807 18:31:06.080456 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.080465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.081588 (+ 1123us) server_negotiation.cc:407] Connection header received
0807 18:31:06.083438 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.083446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.083550 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.083677 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.085092 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.086243 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.087807 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.088640 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.088827 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.092451 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.092462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.092481 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.092531 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.096515 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.097483 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.097489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.097492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.097655 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.098107 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.098112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.098116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.098538 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.098722 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.099481 (+ 759us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.099698 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:31:06.100896 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.080619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35933 (local address 127.0.0.1:54058)
0807 18:31:06.081400 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.081463 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:06.083164 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:06.083866 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:06.083874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:06.084359 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:06.084901 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.084917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.086398 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.086402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.087663 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.087675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.088854 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.088859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.089045 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.089725 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:06.089746 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:06.092198 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:06.096659 (+ 4461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.096666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.096677 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.097298 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.097784 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.097787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.097789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.097977 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.098669 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:06.098674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:06.099006 (+ 332us) client_negotiation.cc:769] Sending connection context
0807 18:31:06.099314 (+ 308us) client_negotiation.cc:240] Negotiation successful
0807 18:31:06.099690 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":353,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.252@KRBTEST.COM" created.
I20260807 18:31:06.102676 9845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:31:06.116663 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.103384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37156)
0807 18:31:06.103550 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.103566 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:06.103645 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:06.104590 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:06.104595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:06.105020 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:06.105150 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.105158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.106517 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.106522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.107393 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.107405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.108255 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.108260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.108499 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.109080 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:06.109097 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:06.111008 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:06.114079 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.114086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.114092 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.114440 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.114816 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.114823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.114828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.114994 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.115667 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:06.115676 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:06.115820 (+ 144us) client_negotiation.cc:769] Sending connection context
0807 18:31:06.115960 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 18:31:06.116234 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":43}
I20260807 18:31:06.116722 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.103670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.27.1.253:41857)
0807 18:31:06.104209 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.104215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.104239 (+ 24us) server_negotiation.cc:407] Connection header received
0807 18:31:06.104279 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.104284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.104442 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.104606 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.105309 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.106343 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.107536 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.108109 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.108248 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.111243 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.111254 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.111266 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.111294 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.113935 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.114579 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.114587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.114590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.114705 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.115104 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.115110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.115113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.115518 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.115740 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.116145 (+ 405us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.116356 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":410,"thread_start_us":266,"threads_started":1}
I20260807 18:31:06.118949 9845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } has no permanent_uuid. Determining permanent_uuid...
I20260807 18:31:06.120339 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.119702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50080)
0807 18:31:06.119957 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.120127 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260807 18:31:06.120657 9787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
I20260807 18:31:06.122349 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.121934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50096)
0807 18:31:06.122106 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.122207 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 18:31:06.122575 9845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:33999: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab.
Entry for principal kudu/127.27.1.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab.
I20260807 18:31:06.134636 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.252:33999
--webserver_interface=127.27.1.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab
--principal=kudu/127.27.1.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:35933,127.27.1.253:41857,127.27.1.252:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:31:06.171916 9845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } attempt: 1
I20260807 18:31:06.173202 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.172646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50108)
0807 18:31:06.172872 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.173015 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 18:31:06.174224 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.173828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50118)
0807 18:31:06.174007 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.174094 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 18:31:06.174456 9845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:33999: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
I20260807 18:31:06.242707 9845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } attempt: 2
I20260807 18:31:06.244117 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.243494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50124)
0807 18:31:06.243731 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.243886 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 18:31:06.245288 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.244808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50134)
0807 18:31:06.244995 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.245109 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 18:31:06.245611 9845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:33999: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
I20260807 18:31:06.269025 9763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } attempt: 5
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:31:06.282397 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.269753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37172)
0807 18:31:06.269954 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.269998 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:06.270142 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:06.270527 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:06.270535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:06.271023 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:06.271301 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.271312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.272323 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.272331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.273216 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.273228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.273971 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.273977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.274168 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.274898 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:06.274916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:06.276940 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:06.279986 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.279993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.279997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.280366 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.280747 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.280753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.280756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.280900 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.281429 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:06.281434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:06.281583 (+ 149us) client_negotiation.cc:769] Sending connection context
0807 18:31:06.281732 (+ 149us) client_negotiation.cc:240] Negotiation successful
0807 18:31:06.282014 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 18:31:06.282395 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.269935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.27.1.253:41857)
0807 18:31:06.270124 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.270130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.270154 (+ 24us) server_negotiation.cc:407] Connection header received
0807 18:31:06.270262 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.270267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.270368 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.270482 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.271486 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.272185 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.273363 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.273824 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.273958 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.277129 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.277133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.277136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.277166 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.279852 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.280492 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.280498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.280502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.280630 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.281013 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.281018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.281021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.281307 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.281464 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.281836 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.282062 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:31:06.283753 9763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } has no permanent_uuid. Determining permanent_uuid...
I20260807 18:31:06.285071 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.284641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50150)
0807 18:31:06.284841 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.284938 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 18:31:06.286080 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.285758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50156)
0807 18:31:06.285903 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.285988 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 18:31:06.286325 9763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:33999: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
I20260807 18:31:06.336884 9845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } attempt: 3
I20260807 18:31:06.337551 9763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } attempt: 1
I20260807 18:31:06.338249 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.337669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50166)
0807 18:31:06.337933 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.338054 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 18:31:06.338630 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.338109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50172)
0807 18:31:06.338297 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.338413 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 18:31:06.339718 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.339337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50180)
0807 18:31:06.339538 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.339621 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260807 18:31:06.339730 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.339225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50174)
0807 18:31:06.339484 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.339578 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260807 18:31:06.339990 9763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:33999: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
W20260807 18:31:06.340003 9845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:33999: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
I20260807 18:31:06.408215 9763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } attempt: 2
I20260807 18:31:06.409380 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.408884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50182)
0807 18:31:06.409128 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.409222 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 18:31:06.410763 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.410443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50192)
0807 18:31:06.410587 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.410657 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 18:31:06.411080 9763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:33999: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
W20260807 18:31:06.472182 9853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:06.472460 9853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:06.472635 9853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:06.479889 9853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:31:06.479986 9853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:06.480064 9853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:06.480166 9853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:31:06.480255 9853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:31:06.483222 9845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } attempt: 4
I20260807 18:31:06.484442 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.483875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50194)
0807 18:31:06.484128 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.484231 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 18:31:06.485462 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.485089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50210)
0807 18:31:06.485241 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.485322 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 18:31:06.485693 9845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:33999: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
I20260807 18:31:06.490830 9853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:35933,127.27.1.253:41857,127.27.1.252:33999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab
--principal=kudu/127.27.1.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.252:33999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.1.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:06.492079 9853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:06.493286 9853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:06.505120 9858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:06.505147 9859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:06.505218 9863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:06.506038 9853 server_base.cc:1034] running on GCE node
I20260807 18:31:06.506762 9853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:06.508118 9853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:06.509354 9853 hybrid_clock.cc:648] HybridClock initialized: now 1786127466509307 us; error 50 us; skew 500 ppm
I20260807 18:31:06.509724 9853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:06.512873 9853 init.cc:377] Logged in from keytab as kudu/127.27.1.252@KRBTEST.COM (short username kudu)
I20260807 18:31:06.514361 9853 webserver.cc:460] Webserver started at http://127.27.1.252:45967/ using document root <none> and password file <none>
I20260807 18:31:06.515074 9853 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:06.515249 9853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:06.515600 9853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:06.517334 9763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } attempt: 3
I20260807 18:31:06.518548 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.517934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50214)
0807 18:31:06.518212 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.518330 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 18:31:06.519264 9853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/data/instance:
uuid: "1c24827762c6438eb6aa900e55ecfa4a"
format_stamp: "Formatted at 2026-08-07 18:31:06 on dist-test-slave-1xrh"
server_key: "f5b73effbe862cc22938114031647c15"
server_key_iv: "66f551f3808aa78382e556efec485155"
server_key_version: "encryptionkey@0"
I20260807 18:31:06.520164 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.519519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50220)
0807 18:31:06.519895 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.520015 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
I20260807 18:31:06.520438 9853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/wal/instance:
uuid: "1c24827762c6438eb6aa900e55ecfa4a"
format_stamp: "Formatted at 2026-08-07 18:31:06 on dist-test-slave-1xrh"
server_key: "f5b73effbe862cc22938114031647c15"
server_key_iv: "66f551f3808aa78382e556efec485155"
server_key_version: "encryptionkey@0"
W20260807 18:31:06.522456 9763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:33999: Network error: Client connection negotiation failed: client connection to 127.27.1.252:33999: connect: Connection refused (error 111)
I20260807 18:31:06.527225 9853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 18:31:06.531226 9870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:06.532634 9853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:31:06.532815 9853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/wal
uuid: "1c24827762c6438eb6aa900e55ecfa4a"
format_stamp: "Formatted at 2026-08-07 18:31:06 on dist-test-slave-1xrh"
server_key: "f5b73effbe862cc22938114031647c15"
server_key_iv: "66f551f3808aa78382e556efec485155"
server_key_version: "encryptionkey@0"
I20260807 18:31:06.533005 9853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:06.547479 9853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:06.551771 9853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:06.552002 9853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:06.570899 9853 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.252:33999
I20260807 18:31:06.570924 9922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.252:33999 every 8 connection(s)
I20260807 18:31:06.572178 9853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
I20260807 18:31:06.573103 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 9853
I20260807 18:31:06.573343 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786127255008662-27655-0/minicluster-data/master-2/wal/instance
I20260807 18:31:06.573933 27655 external_mini_cluster.cc:1428] Setting key df9d14d594ac06e803123b6a1b4e563f
I20260807 18:31:06.577028 9923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:06.583742 9923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:06.596385 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.576052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.27.1.252:33999)
0807 18:31:06.576800 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.576811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.576849 (+ 38us) server_negotiation.cc:407] Connection header received
0807 18:31:06.578456 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.578475 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.578805 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.579197 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.580727 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.582025 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.583411 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.584090 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.584323 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.589221 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.589228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.589238 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.589271 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.592206 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.593136 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.593142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.593147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.593301 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.593700 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.593703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.593704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.594170 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.594507 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.594935 (+ 428us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.595341 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":353,"threads_started":1}
I20260807 18:31:06.601258 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.586416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35933 (local address 127.0.0.1:54064)
0807 18:31:06.587212 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.587252 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:06.587476 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:06.587865 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:06.587874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:06.588521 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:06.588782 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.588798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.589895 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.589899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.590998 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.591014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.591847 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.591854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.592035 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.592607 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:06.592628 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:06.594991 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:06.598616 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.598626 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.598635 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.599095 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.599603 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.599608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.599611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.599822 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.600489 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:06.600495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:06.600634 (+ 139us) client_negotiation.cc:769] Sending connection context
0807 18:31:06.600768 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 18:31:06.600972 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"spinlock_wait_cycles":12544,"thread_start_us":378,"threads_started":1}
I20260807 18:31:06.601557 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.586389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.27.1.254:35933)
0807 18:31:06.586565 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.586570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.587348 (+ 778us) server_negotiation.cc:407] Connection header received
0807 18:31:06.587621 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.587625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.587688 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.587767 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.588963 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.589718 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.591180 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.591644 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.591788 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.595224 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.595231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.595236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.595278 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.598474 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.599266 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.599270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.599273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.599452 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.600000 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.600006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.600009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.600326 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.600501 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.600944 (+ 443us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.601185 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 18:31:06.602839 9923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:31:06.614634 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.599460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.27.1.254:35933)
0807 18:31:06.600043 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.600047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.600065 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:31:06.600103 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.600106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.600156 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.600233 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.601864 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.602844 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.604420 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.604817 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.604953 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.609117 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.609122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.609125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.609154 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.612013 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.612637 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.612644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.612647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.612767 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.613154 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.613159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.613163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.613446 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.613716 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.614011 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.614221 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":247,"threads_started":1}
I20260807 18:31:06.615440 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.603764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.27.1.253:41857)
0807 18:31:06.603977 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.603982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.603997 (+ 15us) server_negotiation.cc:407] Connection header received
0807 18:31:06.604046 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.604049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.604104 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.604214 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.604904 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.605532 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.606683 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.607115 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.607304 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.610127 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.610133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.610139 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.610177 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.612780 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.613419 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.613425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.613429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.613580 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.613970 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.613977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.613981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.614219 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.614378 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.614919 (+ 541us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.615110 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 18:31:06.616379 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.603514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:41857 (local address 127.0.0.1:37186)
0807 18:31:06.603647 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.603664 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:06.603751 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:06.604238 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:06.604242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:06.604610 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:06.604742 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.604750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.605675 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.605682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.606541 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.606553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.607314 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.607319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.607561 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.608044 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:06.608066 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:06.609933 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:06.612926 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.612931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.612934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.613299 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.613698 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.613703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.613707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.613857 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.614344 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:06.614348 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:06.614606 (+ 258us) client_negotiation.cc:769] Sending connection context
0807 18:31:06.615753 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0807 18:31:06.616061 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260807 18:31:06.617377 9923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:31:06.631016 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.618086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50242)
0807 18:31:06.618264 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.618282 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:06.618376 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:06.618664 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:06.618666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:06.618984 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:06.619115 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.619170 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.620162 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.620168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.621263 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.621275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.622183 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.622189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.622399 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.622893 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:06.622920 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:06.625434 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:06.628500 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.628506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.628510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.628998 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.629358 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.629363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.629367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.629542 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.630120 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:06.630123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:06.630242 (+ 119us) client_negotiation.cc:769] Sending connection context
0807 18:31:06.630350 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 18:31:06.630667 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 18:31:06.631536 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.618248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.27.1.252:33999)
0807 18:31:06.618394 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.618400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.618415 (+ 15us) server_negotiation.cc:407] Connection header received
0807 18:31:06.618472 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.618478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.618549 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.618648 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.619300 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.620032 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.621419 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.621990 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.622114 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.625619 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.625628 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.625636 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.625681 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.628352 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.629109 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.629113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.629115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.629233 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.629653 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.629657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.629659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.629995 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.630139 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.630492 (+ 353us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.630874 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":36}
I20260807 18:31:06.632529 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.616301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.27.1.253:41857)
0807 18:31:06.616469 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.616477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.616496 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:31:06.616739 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.616747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.616802 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.616912 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.618418 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.619353 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.620884 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.621270 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.621394 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.626521 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.626530 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.626537 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.626576 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.629740 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.630505 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.630510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.630513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.630673 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.631102 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.631108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.631111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.631420 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.632019 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.632119 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.632274 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 18:31:06.637094 9923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a: Bootstrap starting.
I20260807 18:31:06.637681 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:31:06.641299 9923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:06.642783 9923 log.cc:824] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:06.646555 9923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a: No bootstrap required, opened a new log
I20260807 18:31:06.660391 9923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } }
I20260807 18:31:06.660867 9923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:06.661006 9923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c24827762c6438eb6aa900e55ecfa4a, State: Initialized, Role: FOLLOWER
I20260807 18:31:06.661787 9923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } }
I20260807 18:31:06.667110 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.643882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.27.1.252:33999)
0807 18:31:06.644137 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.644145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.644985 (+ 840us) server_negotiation.cc:407] Connection header received
0807 18:31:06.645249 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.645255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.645331 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.645413 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.648546 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.650064 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.653124 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.654191 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.654329 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.659543 (+ 5214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.659549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.659554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.659589 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.663516 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.664478 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.664486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.664492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.664666 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.665179 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.665187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.665191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.665599 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.665852 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.666501 (+ 649us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.666725 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:31:06.668828 9938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } } }
I20260807 18:31:06.669201 9938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:31:06.671049 9923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:31:06.674407 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.645072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.27.1.253:41857)
0807 18:31:06.645532 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.645540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.656060 (+ 10520us) server_negotiation.cc:407] Connection header received
0807 18:31:06.656229 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.656237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.656319 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.656419 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.659018 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.660119 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.662441 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.663143 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.663245 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.666917 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.666936 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.666942 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.666986 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.670648 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.671482 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.671490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.671495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.671648 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.672215 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.672221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.672227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.672635 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.673697 (+ 1062us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.673832 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.674057 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342}
I20260807 18:31:06.676065 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.643728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.27.1.254:35933)
0807 18:31:06.643952 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.643960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.655971 (+ 12011us) server_negotiation.cc:407] Connection header received
0807 18:31:06.656179 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.656185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.656269 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.656338 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.658760 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.659837 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.661948 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.662725 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.662859 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.667666 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.667675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.667683 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.667739 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.671934 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.672841 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.672847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.672850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.673010 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.673629 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.673633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.673637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.673993 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.674209 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.675364 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.675593 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260807 18:31:06.681209 9949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 18:31:06.681339 9949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 18:31:06.678726 9853 master_runner.cc:186] Error getting master registration for 127.27.1.254:35933: OK, instance_id { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" instance_seqno: 1786127465556739 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 18:31:06.687443 9853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 18:31:06.699671 9763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } attempt: 4
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
I20260807 18:31:06.714387 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.700231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50266)
0807 18:31:06.700434 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.700454 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:06.700562 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:06.700916 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:06.700921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:06.701346 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:06.701504 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.701513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.702585 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.702592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.704158 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.704173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.705171 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.705175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.705294 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.705920 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:06.705940 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:06.708018 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:06.711849 (+ 3831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.711853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.711856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.712212 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.712642 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.712645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.712647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.712783 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.713381 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:06.713384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:06.713545 (+ 161us) client_negotiation.cc:769] Sending connection context
0807 18:31:06.713697 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 18:31:06.714007 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 18:31:06.715067 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.700321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.27.1.252:33999)
0807 18:31:06.700472 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.700479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.700499 (+ 20us) server_negotiation.cc:407] Connection header received
0807 18:31:06.700694 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.700701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.700784 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.700890 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.701648 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.702450 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.704317 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.705019 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.705161 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.708201 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.708208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.708218 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.708255 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.711721 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.712330 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.712337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.712343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.712516 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.712894 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.712899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.712902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.713263 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.713415 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.713810 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.714051 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 18:31:06.719713 9763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73: Bootstrap starting.
I20260807 18:31:06.722759 9763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:06.723892 9763 log.cc:824] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:06.727257 9763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73: No bootstrap required, opened a new log
I20260807 18:31:06.738232 9763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } }
I20260807 18:31:06.738619 9763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:06.738745 9763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f02ed4351749d8aaeec8a320526a73, State: Initialized, Role: FOLLOWER
I20260807 18:31:06.739413 9763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } }
I20260807 18:31:06.740376 9952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } } }
I20260807 18:31:06.740777 9952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:31:06.741706 9763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 18:31:06.754097 9964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 18:31:06.754237 9964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 18:31:06.752147 9692 master_runner.cc:186] Error getting master registration for 127.27.1.253:41857: OK, instance_id { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" instance_seqno: 1786127466060418 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 18:31:06.760176 9692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 18:31:06.784947 9845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } attempt: 5
Aug 07 18:31:06 dist-test-slave-1xrh krb5kdc[9677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
W20260807 18:31:06.796168 9716 tablet.cc:2369] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:06.800269 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.785593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:33999 (local address 127.0.0.1:50270)
0807 18:31:06.785855 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:06.785876 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:06.786004 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:06.786349 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:06.786353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:06.786815 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:06.787017 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.787026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.788072 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.788078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.788959 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:06.788970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.789726 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.789732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:06.789940 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.790558 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:06.790576 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:06.792875 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:06.797706 (+ 4831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.797712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.797716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.798095 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.798524 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:06.798531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:06.798534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:06.798694 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:06.799360 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:06.799364 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:06.799573 (+ 209us) client_negotiation.cc:769] Sending connection context
0807 18:31:06.799740 (+ 167us) client_negotiation.cc:240] Negotiation successful
0807 18:31:06.800009 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260807 18:31:06.801118 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:06.785678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.27.1.252:33999)
0807 18:31:06.785858 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:06.785865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:06.785946 (+ 81us) server_negotiation.cc:407] Connection header received
0807 18:31:06.786135 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:06.786140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:06.786206 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:06.786320 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:06.787188 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.787916 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.789102 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:06.789560 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:06.789698 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:06.793091 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:06.793097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:06.793102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:06.793143 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:06.797552 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.798220 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.798227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.798230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.798418 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:06.798854 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:06.798872 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:06.798875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:06.799244 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:06.799392 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:06.800578 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0807 18:31:06.800844 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260807 18:31:06.803923 9877 tablet.cc:2369] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:06.805886 9845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913: Bootstrap starting.
I20260807 18:31:06.809335 9845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:06.810671 9845 log.cc:824] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:06.814714 9845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913: No bootstrap required, opened a new log
I20260807 18:31:06.821859 9845 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } }
I20260807 18:31:06.822184 9845 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:06.822324 9845 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e2941c3cbd423faeec4af3c76b9913, State: Initialized, Role: FOLLOWER
I20260807 18:31:06.822918 9845 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } }
I20260807 18:31:06.823910 9967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } } }
I20260807 18:31:06.824339 9967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:31:06.825073 9845 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:31:06.845880 9770 master_runner.cc:263] No leader master found from master a6e2941c3cbd423faeec4af3c76b9913
I20260807 18:31:06.846083 9770 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 18:31:06.849586 9979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 18:31:06.849819 9979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 18:31:07.016670 9938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:31:07.017032 9938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } }
I20260807 18:31:07.018878 9938 leader_election.cc:288] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12f02ed4351749d8aaeec8a320526a73 (127.27.1.254:35933), a6e2941c3cbd423faeec4af3c76b9913 (127.27.1.253:41857)
I20260807 18:31:07.020476 9814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c24827762c6438eb6aa900e55ecfa4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e2941c3cbd423faeec4af3c76b9913" is_pre_election: true
I20260807 18:31:07.021167 9814 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c24827762c6438eb6aa900e55ecfa4a in term 0.
I20260807 18:31:07.022575 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c24827762c6438eb6aa900e55ecfa4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f02ed4351749d8aaeec8a320526a73" is_pre_election: true
I20260807 18:31:07.023483 9873 leader_election.cc:302] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c24827762c6438eb6aa900e55ecfa4a, a6e2941c3cbd423faeec4af3c76b9913; no voters:
I20260807 18:31:07.023988 9737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c24827762c6438eb6aa900e55ecfa4a in term 0.
I20260807 18:31:07.024946 9938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:31:07.025098 9938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:31:07.025224 9938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:07.028880 9938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } }
I20260807 18:31:07.030501 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c24827762c6438eb6aa900e55ecfa4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f02ed4351749d8aaeec8a320526a73"
I20260807 18:31:07.030740 9737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:07.035594 9737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c24827762c6438eb6aa900e55ecfa4a in term 1.
I20260807 18:31:07.036424 9873 leader_election.cc:302] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f02ed4351749d8aaeec8a320526a73, 1c24827762c6438eb6aa900e55ecfa4a; no voters:
I20260807 18:31:07.038584 9938 leader_election.cc:288] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [CANDIDATE]: Term 1 election: Requested vote from peers 12f02ed4351749d8aaeec8a320526a73 (127.27.1.254:35933), a6e2941c3cbd423faeec4af3c76b9913 (127.27.1.253:41857)
I20260807 18:31:07.039808 9938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:31:07.039819 9814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c24827762c6438eb6aa900e55ecfa4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e2941c3cbd423faeec4af3c76b9913"
I20260807 18:31:07.040131 9814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:07.041036 9938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [term 1 LEADER]: Becoming Leader. State: Replica: 1c24827762c6438eb6aa900e55ecfa4a, State: Running, Role: LEADER
I20260807 18:31:07.041998 9938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } }
I20260807 18:31:07.043799 9814 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c24827762c6438eb6aa900e55ecfa4a in term 1.
W20260807 18:31:07.050174 9791 tablet.cc:2369] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:07.052604 9981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c24827762c6438eb6aa900e55ecfa4a. Latest consensus state: current_term: 1 leader_uuid: "1c24827762c6438eb6aa900e55ecfa4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } } }
I20260807 18:31:07.053085 9981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:07.065815 9983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:31:07.068298 9983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:31:07.084664 9814 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [term 1 FOLLOWER]: Refusing update from remote peer 1c24827762c6438eb6aa900e55ecfa4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:31:07.085803 9938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:31:07.093451 9737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [term 1 FOLLOWER]: Refusing update from remote peer 1c24827762c6438eb6aa900e55ecfa4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:31:07.094468 9938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:31:07.114216 9967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c24827762c6438eb6aa900e55ecfa4a. Latest consensus state: current_term: 1 leader_uuid: "1c24827762c6438eb6aa900e55ecfa4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } } }
I20260807 18:31:07.114552 9967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:31:07.118362 9984 mvcc.cc:205] Tried to move back new op lower bound from 7315978105167925248 to 7315978105010765824. Current Snapshot: MvccSnapshot[applied={T|T < 7315978105167925248}]
I20260807 18:31:07.123063 9952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c24827762c6438eb6aa900e55ecfa4a. Latest consensus state: current_term: 1 leader_uuid: "1c24827762c6438eb6aa900e55ecfa4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } } }
I20260807 18:31:07.123399 9952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:31:07.126400 9986 mvcc.cc:205] Tried to move back new op lower bound from 7315978105167925248 to 7315978105010765824. Current Snapshot: MvccSnapshot[applied={T|T < 7315978105167925248}]
I20260807 18:31:07.131258 9967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c24827762c6438eb6aa900e55ecfa4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } } }
I20260807 18:31:07.131533 9967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:31:07.132560 9938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c24827762c6438eb6aa900e55ecfa4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } } }
I20260807 18:31:07.132917 9938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:07.136626 9983 catalog_manager.cc:1347] Generated new cluster ID: 9cbae3c9bcc842958af265c881996ff2
I20260807 18:31:07.136747 9983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:31:07.139339 9981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c24827762c6438eb6aa900e55ecfa4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } } }
I20260807 18:31:07.139992 9981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:07.147534 9952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c24827762c6438eb6aa900e55ecfa4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f02ed4351749d8aaeec8a320526a73" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35933 } } peers { permanent_uuid: "a6e2941c3cbd423faeec4af3c76b9913" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 41857 } } peers { permanent_uuid: "1c24827762c6438eb6aa900e55ecfa4a" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 33999 } } }
I20260807 18:31:07.147804 9952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:31:07.168399 9983 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:31:07.170202 9983 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:31:07.188804 9983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a: Generated new TSK 0
I20260807 18:31:07.189895 9983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:31:07.756230 9964 catalog_manager.cc:1259] Loaded cluster ID: 9cbae3c9bcc842958af265c881996ff2
I20260807 18:31:07.756361 9964 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73: loading cluster ID for follower catalog manager: success
I20260807 18:31:07.758795 9964 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73: acquiring CA information for follower catalog manager: success
I20260807 18:31:07.760568 9964 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12f02ed4351749d8aaeec8a320526a73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 18:31:07.851972 9979 catalog_manager.cc:1259] Loaded cluster ID: 9cbae3c9bcc842958af265c881996ff2
I20260807 18:31:07.852084 9979 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913: loading cluster ID for follower catalog manager: success
I20260807 18:31:07.854377 9979 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913: acquiring CA information for follower catalog manager: success
I20260807 18:31:07.856088 9979 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6e2941c3cbd423faeec4af3c76b9913: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 18:31:08.291401 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 9692
I20260807 18:31:08.293139 9758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:08.406327 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 9692
I20260807 18:31:08.431876 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 9770
I20260807 18:31:08.433106 9839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:08.486588 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:08.485788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35933 (local address 127.0.0.1:54096)
0807 18:31:08.486299 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:08.486414 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:35933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":292,"threads_started":1}
W20260807 18:31:08.486910 9873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.1.254:35933: connect: Connection refused (error 111)
I20260807 18:31:08.488353 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:08.488010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35933 (local address 127.0.0.1:54100)
0807 18:31:08.488166 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:08.488245 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:35933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 18:31:08.488546 9873 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1c24827762c6438eb6aa900e55ecfa4a -> Peer 12f02ed4351749d8aaeec8a320526a73 (127.27.1.254:35933): Couldn't send request to peer 12f02ed4351749d8aaeec8a320526a73. Status: Network error: Client connection negotiation failed: client connection to 127.27.1.254:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 18:31:08.542817 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 9770
I20260807 18:31:08.567036 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 9853
I20260807 18:31:08.568221 9918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:08.677977 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 9853
2026-08-07T18:31:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5050 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:31:08 dist-test-slave-1xrh krb5kdc[10004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:31:08 dist-test-slave-1xrh krb5kdc[10004](info): set up 2 sockets
Aug 07 18:31:08 dist-test-slave-1xrh krb5kdc[10004](info): commencing operation
krb5kdc: starting...
W20260807 18:31:10.049738 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.327s user 0.006s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:31:10 dist-test-slave-1xrh krb5kdc[10004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:31:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:31:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:31:10.327955 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:42647
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:42647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:10.666150 10019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:10.666422 10019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:10.666611 10019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:10.673702 10019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:31:10.673799 10019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:10.673888 10019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:10.673995 10019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:31:10.674080 10019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:31:10.684229 10019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:42647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:42647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:10.685585 10019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:10.686901 10019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:10.698619 10029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:10.698676 10027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:10.698699 10026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:10.699462 10019 server_base.cc:1034] running on GCE node
I20260807 18:31:10.700153 10019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:10.701434 10019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:10.702646 10019 hybrid_clock.cc:648] HybridClock initialized: now 1786127470702591 us; error 44 us; skew 500 ppm
I20260807 18:31:10.702935 10019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:10 dist-test-slave-1xrh krb5kdc[10004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127470, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:10.706077 10019 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:31:10.707541 10019 webserver.cc:460] Webserver started at http://127.27.1.254:37147/ using document root <none> and password file <none>
I20260807 18:31:10.708303 10019 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:10.708431 10019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:10.708755 10019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:10.711688 10019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "5811379317654381a40ea64796b0cb3b"
format_stamp: "Formatted at 2026-08-07 18:31:10 on dist-test-slave-1xrh"
server_key: "382dc27924587941775124147dd5ea66"
server_key_iv: "b97f722667f97059578ef553bf564da7"
server_key_version: "encryptionkey@0"
I20260807 18:31:10.712440 10019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "5811379317654381a40ea64796b0cb3b"
format_stamp: "Formatted at 2026-08-07 18:31:10 on dist-test-slave-1xrh"
server_key: "382dc27924587941775124147dd5ea66"
server_key_iv: "b97f722667f97059578ef553bf564da7"
server_key_version: "encryptionkey@0"
I20260807 18:31:10.717454 10019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 18:31:10.720872 10036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:10.722198 10019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 18:31:10.722394 10019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "5811379317654381a40ea64796b0cb3b"
format_stamp: "Formatted at 2026-08-07 18:31:10 on dist-test-slave-1xrh"
server_key: "382dc27924587941775124147dd5ea66"
server_key_iv: "b97f722667f97059578ef553bf564da7"
server_key_version: "encryptionkey@0"
I20260807 18:31:10.722589 10019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:10.739449 10019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:10.743463 10019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:10.743752 10019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:10.762804 10019 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:42647
I20260807 18:31:10.762837 10088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:42647 every 8 connection(s)
I20260807 18:31:10.764187 10019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:31:10.766795 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 10019
I20260807 18:31:10.767038 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:31:10.767763 27655 external_mini_cluster.cc:1428] Setting key 1207e8530e72536b5d7b0e3e57ffc04c
I20260807 18:31:10.767958 10089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:10.782399 10089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b: Bootstrap starting.
Aug 07 18:31:10 dist-test-slave-1xrh krb5kdc[10004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:10.785561 10089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:10.786696 10089 log.cc:824] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:10.789507 10089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b: No bootstrap required, opened a new log
I20260807 18:31:10.791514 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:10.769636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.27.1.254:42647)
0807 18:31:10.770409 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:10.770421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:10.770466 (+ 45us) server_negotiation.cc:407] Connection header received
0807 18:31:10.772504 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:10.772526 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:10.772907 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:10.773231 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:10.775063 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:10.776387 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:10.778191 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:10.778853 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:10.778984 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:10.784094 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:10.784101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:10.784117 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:10.784152 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:10.787515 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:10.788227 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:10.788240 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:10.788246 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:10.788406 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:10.788796 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:10.788801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:10.788804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:10.789276 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:10.789609 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:10.790062 (+ 453us) server_negotiation.cc:299] Negotiation successful
0807 18:31:10.790489 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":346,"threads_started":1}
I20260807 18:31:10.795469 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:31:10.798233 10089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811379317654381a40ea64796b0cb3b" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42647 } }
I20260807 18:31:10.798671 10089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:10.798782 10089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5811379317654381a40ea64796b0cb3b, State: Initialized, Role: FOLLOWER
I20260807 18:31:10.799305 10089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811379317654381a40ea64796b0cb3b" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42647 } }
I20260807 18:31:10.799515 10089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:31:10.799611 10089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:31:10.799726 10089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:10.801745 10089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811379317654381a40ea64796b0cb3b" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42647 } }
I20260807 18:31:10.802155 10089 leader_election.cc:302] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5811379317654381a40ea64796b0cb3b; no voters:
I20260807 18:31:10.802673 10089 leader_election.cc:288] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:31:10.802815 10094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:31:10.803556 10094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [term 1 LEADER]: Becoming Leader. State: Replica: 5811379317654381a40ea64796b0cb3b, State: Running, Role: LEADER
I20260807 18:31:10.804164 10094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811379317654381a40ea64796b0cb3b" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42647 } }
I20260807 18:31:10.804765 10089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:31:10.807436 10096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5811379317654381a40ea64796b0cb3b. Latest consensus state: current_term: 1 leader_uuid: "5811379317654381a40ea64796b0cb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811379317654381a40ea64796b0cb3b" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42647 } } }
I20260807 18:31:10.807857 10096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:10.808389 10095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5811379317654381a40ea64796b0cb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811379317654381a40ea64796b0cb3b" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 42647 } } }
I20260807 18:31:10.808676 10095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:10.809495 10106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:31:10.813649 10106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:31:10.821977 10106 catalog_manager.cc:1347] Generated new cluster ID: 04c8e5761b7745cba6c68d07028e6bb1
I20260807 18:31:10.822094 10106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:31:10.835743 10106 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:31:10.836956 10106 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:31:10.842964 10106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5811379317654381a40ea64796b0cb3b: Generated new TSK 0
I20260807 18:31:10.843750 10106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:31:10.855731 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 10019
I20260807 18:31:10.856864 10084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:10.947716 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 10019
2026-08-07T18:31:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2265 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:31:11 dist-test-slave-1xrh krb5kdc[10118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:31:11 dist-test-slave-1xrh krb5kdc[10118](info): set up 2 sockets
Aug 07 18:31:11 dist-test-slave-1xrh krb5kdc[10118](info): commencing operation
krb5kdc: starting...
W20260807 18:31:12.165288 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:31:12 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:31:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:31:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:31:12.442385 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:33219
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:33219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:12.773171 10131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:12.773480 10131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:12.773620 10131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:12.781211 10131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:31:12.781320 10131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:12.781469 10131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:12.781571 10131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:31:12.781659 10131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:31:12.791450 10131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:33219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:33219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:12.792817 10131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:12.794138 10131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:12.806247 10139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:12.806330 10137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:12.806383 10136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:12.807201 10131 server_base.cc:1034] running on GCE node
I20260807 18:31:12.807935 10131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:12.809317 10131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:12.810572 10131 hybrid_clock.cc:648] HybridClock initialized: now 1786127472810572 us; error 101 us; skew 500 ppm
I20260807 18:31:12.810937 10131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:12 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127472, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:12.814257 10131 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:31:12.815757 10131 webserver.cc:460] Webserver started at http://127.27.1.254:46787/ using document root <none> and password file <none>
I20260807 18:31:12.816448 10131 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:12.816599 10131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:12.816903 10131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:12.819828 10131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45"
format_stamp: "Formatted at 2026-08-07 18:31:12 on dist-test-slave-1xrh"
server_key: "8aede755105c6f73773c4d05622f9835"
server_key_iv: "bea453d3e7aa58eb8a015774011c983e"
server_key_version: "encryptionkey@0"
I20260807 18:31:12.820581 10131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45"
format_stamp: "Formatted at 2026-08-07 18:31:12 on dist-test-slave-1xrh"
server_key: "8aede755105c6f73773c4d05622f9835"
server_key_iv: "bea453d3e7aa58eb8a015774011c983e"
server_key_version: "encryptionkey@0"
I20260807 18:31:12.825618 10131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:31:12.828869 10146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:12.830106 10131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 18:31:12.830283 10131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45"
format_stamp: "Formatted at 2026-08-07 18:31:12 on dist-test-slave-1xrh"
server_key: "8aede755105c6f73773c4d05622f9835"
server_key_iv: "bea453d3e7aa58eb8a015774011c983e"
server_key_version: "encryptionkey@0"
I20260807 18:31:12.830484 10131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:12.850809 10131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:12.855254 10131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:12.855548 10131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:12.874238 10131 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:33219
I20260807 18:31:12.874280 10201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:33219 every 8 connection(s)
I20260807 18:31:12.875670 10131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:31:12.878412 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 10131
I20260807 18:31:12.878649 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:31:12.879354 27655 external_mini_cluster.cc:1428] Setting key a0c7cd7f3a7645595d16672f4805b21f
I20260807 18:31:12.879405 10202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:12.894052 10202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Bootstrap starting.
Aug 07 18:31:12 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:12.897161 10202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:12.898267 10202 log.cc:824] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:12.901048 10202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: No bootstrap required, opened a new log
I20260807 18:31:12.902536 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:12.881313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.27.1.254:33219)
0807 18:31:12.882174 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:12.882188 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:12.882231 (+ 43us) server_negotiation.cc:407] Connection header received
0807 18:31:12.884050 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:12.884070 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:12.884431 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:12.884744 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:12.886645 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:12.887997 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:12.889856 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:12.890535 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:12.890683 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:12.895578 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:12.895585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:12.895597 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:12.895638 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:12.898739 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:12.899509 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:12.899516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:12.899521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:12.899687 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:12.900059 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:12.900064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:12.900069 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:12.900544 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:12.900884 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:12.901235 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 18:31:12.901635 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":377,"threads_started":1}
I20260807 18:31:12.909456 10202 raft_consensus.cc:348] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:12.909911 10202 raft_consensus.cc:374] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:12.910022 10202 raft_consensus.cc:729] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffdcd7fda3943d8a7ec70e1c4d26d45, State: Initialized, Role: FOLLOWER
I20260807 18:31:12.910676 10202 consensus_queue.cc:260] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:12.910983 10202 raft_consensus.cc:388] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:31:12.911159 10202 raft_consensus.cc:482] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:31:12.911347 10202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:12.914234 10202 raft_consensus.cc:504] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:12.914721 10202 leader_election.cc:302] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bffdcd7fda3943d8a7ec70e1c4d26d45; no voters:
I20260807 18:31:12.915371 10202 leader_election.cc:288] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:31:12.915481 10207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:31:12.916162 10207 raft_consensus.cc:686] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 1 LEADER]: Becoming Leader. State: Replica: bffdcd7fda3943d8a7ec70e1c4d26d45, State: Running, Role: LEADER
I20260807 18:31:12.916734 10207 consensus_queue.cc:237] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:12.917389 10202 sys_catalog.cc:564] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:31:12.920054 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } } }
I20260807 18:31:12.920032 10209 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bffdcd7fda3943d8a7ec70e1c4d26d45. Latest consensus state: current_term: 1 leader_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } } }
I20260807 18:31:12.920449 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:12.920498 10209 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:12.921438 10217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:31:12.927143 10217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:31:12.935878 10217 catalog_manager.cc:1347] Generated new cluster ID: 2a3feffc53304f7faf53facd6d05de3d
I20260807 18:31:12.936012 10217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:31:12.957861 10217 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:31:12.959823 10217 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:31:12.970177 10217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Generated new TSK 0
I20260807 18:31:12.970995 10217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:31:13.055563 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:33219
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:13.411571 10229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:13.411922 10229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:13.412081 10229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:13.419674 10229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:13.419823 10229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:13.419998 10229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:31:13.429976 10229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:13.431375 10229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:13.432685 10229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:13.446552 10235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:13.448457 10238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:13.448921 10229 server_base.cc:1034] running on GCE node
W20260807 18:31:13.451392 10234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:13.452337 10229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:13.453501 10229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:13.454764 10229 hybrid_clock.cc:648] HybridClock initialized: now 1786127473454642 us; error 117 us; skew 500 ppm
I20260807 18:31:13.455106 10229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:13 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:13.458745 10229 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:31:13.460669 10229 webserver.cc:460] Webserver started at http://127.27.1.193:37701/ using document root <none> and password file <none>
I20260807 18:31:13.461488 10229 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:13.461654 10229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:13.462006 10229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:13.465483 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "d4299a843b4049ceaa60048e64bebc80"
format_stamp: "Formatted at 2026-08-07 18:31:13 on dist-test-slave-1xrh"
server_key: "255a46d1e022f89df9cf3600ca79d5d6"
server_key_iv: "d9c8fd1d5b59f95e0064889fc647e497"
server_key_version: "encryptionkey@0"
I20260807 18:31:13.466449 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "d4299a843b4049ceaa60048e64bebc80"
format_stamp: "Formatted at 2026-08-07 18:31:13 on dist-test-slave-1xrh"
server_key: "255a46d1e022f89df9cf3600ca79d5d6"
server_key_iv: "d9c8fd1d5b59f95e0064889fc647e497"
server_key_version: "encryptionkey@0"
I20260807 18:31:13.472272 10229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 18:31:13.475888 10246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:13.477224 10229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:31:13.477420 10229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "d4299a843b4049ceaa60048e64bebc80"
format_stamp: "Formatted at 2026-08-07 18:31:13 on dist-test-slave-1xrh"
server_key: "255a46d1e022f89df9cf3600ca79d5d6"
server_key_iv: "d9c8fd1d5b59f95e0064889fc647e497"
server_key_version: "encryptionkey@0"
I20260807 18:31:13.477651 10229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:13.498070 10229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:13.501999 10229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:13.502293 10229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:13.503310 10229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:13.504674 10229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:13.504776 10229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:13.504923 10229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:13.505000 10229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:13.539294 10229 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:44151
I20260807 18:31:13.539381 10365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:44151 every 8 connection(s)
I20260807 18:31:13.540616 10229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:31:13.545652 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 10229
I20260807 18:31:13.545879 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:31:13.546720 27655 external_mini_cluster.cc:1428] Setting key 0f706cfbca08d2b7d3e51c2ae053fffc
Aug 07 18:31:13 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:13.563436 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:13.543210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:54137 (local address 127.27.1.254:33219)
0807 18:31:13.543693 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:13.543697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:13.544499 (+ 802us) server_negotiation.cc:407] Connection header received
0807 18:31:13.547076 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:13.547081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:13.547204 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:13.547303 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:13.549106 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:13.549812 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:13.552369 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:13.552774 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:13.552877 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:13.557100 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:13.557105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:13.557108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:13.557141 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:13.560071 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:13.560886 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:13.560898 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:13.560901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:13.561028 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:13.561520 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:13.561525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:13.561527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:13.561815 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:13.561986 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:13.562756 (+ 770us) server_negotiation.cc:299] Negotiation successful
0807 18:31:13.563015 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":263,"threads_started":1}
I20260807 18:31:13.564482 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:13.543628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:54137)
0807 18:31:13.544335 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:13.544374 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:13.546735 (+ 2361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:13.547564 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:13.547572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:13.548188 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:13.548911 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:13.548946 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:13.549989 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:13.549998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:13.552212 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:13.552231 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:13.553020 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:13.553029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:13.553353 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:13.554094 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:13.554134 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:13.556851 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:13.560241 (+ 3390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:13.560251 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:13.560274 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:13.560730 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:13.561150 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:13.561157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:13.561161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:13.561379 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:13.561958 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:13.561967 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:13.562326 (+ 359us) client_negotiation.cc:769] Sending connection context
0807 18:31:13.562615 (+ 289us) client_negotiation.cc:240] Negotiation successful
0807 18:31:13.563195 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":293,"threads_started":1}
I20260807 18:31:13.566206 10366 heartbeater.cc:347] Connected to a master server at 127.27.1.254:33219
I20260807 18:31:13.566592 10366 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:13.567533 10366 heartbeater.cc:511] Master 127.27.1.254:33219 requested a full tablet report, sending...
I20260807 18:31:13.570067 10167 ts_manager.cc:194] Registered new tserver with Master: d4299a843b4049ceaa60048e64bebc80 (127.27.1.193:44151)
I20260807 18:31:13.572307 10167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:54137
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:31:13.612435 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:33219
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:13.946969 10372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:13.947311 10372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:13.947476 10372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:13.954741 10372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:13.954854 10372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:13.955017 10372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:31:13.965353 10372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:13.966645 10372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:13.968112 10372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:13.981513 10378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:13.981601 10380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:13.981805 10377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:13.982550 10372 server_base.cc:1034] running on GCE node
I20260807 18:31:13.983237 10372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:13.984238 10372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:13.985486 10372 hybrid_clock.cc:648] HybridClock initialized: now 1786127473985440 us; error 60 us; skew 500 ppm
I20260807 18:31:13.985802 10372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:13 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:13.995334 10372 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:31:13.996937 10372 webserver.cc:460] Webserver started at http://127.27.1.194:38507/ using document root <none> and password file <none>
I20260807 18:31:13.997663 10372 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:13.997825 10372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:13.998154 10372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:14.001161 10372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "841c868a39524c218efc67bbc5e46a53"
format_stamp: "Formatted at 2026-08-07 18:31:13 on dist-test-slave-1xrh"
server_key: "e13dd5afc85e33d14a7d3289deea9ca6"
server_key_iv: "aadd88ec397d409715ea30109c7385b8"
server_key_version: "encryptionkey@0"
I20260807 18:31:14.001952 10372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "841c868a39524c218efc67bbc5e46a53"
format_stamp: "Formatted at 2026-08-07 18:31:13 on dist-test-slave-1xrh"
server_key: "e13dd5afc85e33d14a7d3289deea9ca6"
server_key_iv: "aadd88ec397d409715ea30109c7385b8"
server_key_version: "encryptionkey@0"
I20260807 18:31:14.007189 10372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 18:31:14.010582 10388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:14.011900 10372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:31:14.012074 10372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "841c868a39524c218efc67bbc5e46a53"
format_stamp: "Formatted at 2026-08-07 18:31:13 on dist-test-slave-1xrh"
server_key: "e13dd5afc85e33d14a7d3289deea9ca6"
server_key_iv: "aadd88ec397d409715ea30109c7385b8"
server_key_version: "encryptionkey@0"
I20260807 18:31:14.012284 10372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:14.035898 10372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:14.040366 10372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:14.040683 10372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:14.041689 10372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:14.043022 10372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:14.043124 10372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:14.043289 10372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:14.043363 10372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:14.076718 10372 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:34141
I20260807 18:31:14.076774 10506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:34141 every 8 connection(s)
I20260807 18:31:14.078037 10372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:31:14.079522 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 10372
I20260807 18:31:14.079733 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:31:14.080202 27655 external_mini_cluster.cc:1428] Setting key cb17ff85e27419fb605718a3f4c0b68c
Aug 07 18:31:14 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:14.101738 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.080706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:48507 (local address 127.27.1.254:33219)
0807 18:31:14.083178 (+ 2472us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:14.083184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:14.083202 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:31:14.084767 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:14.084771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:14.084836 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:14.084918 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:14.086979 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.087752 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.090402 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.090811 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.090914 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.095167 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:14.095177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:14.095184 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:14.095231 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:14.098373 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:14.099341 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:14.099345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:14.099347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:14.099472 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:14.099988 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:14.099992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:14.099994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:14.100247 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:14.100357 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:14.101314 (+ 957us) server_negotiation.cc:299] Negotiation successful
0807 18:31:14.101476 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2336,"thread_start_us":2216,"threads_started":1}
I20260807 18:31:14.103323 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.081385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:48507)
0807 18:31:14.082245 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:14.082283 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:14.084465 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:14.085204 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:14.085216 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:14.085989 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:14.086798 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.086827 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.087902 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.087910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.090144 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.090164 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.091082 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.091092 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.091450 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.092197 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:14.092233 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:14.094902 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:14.098521 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:14.098533 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:14.098552 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:14.099229 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:14.099584 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:14.099604 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:14.099610 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:14.099865 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:14.100388 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:14.100399 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:14.100864 (+ 465us) client_negotiation.cc:769] Sending connection context
0807 18:31:14.101218 (+ 354us) client_negotiation.cc:240] Negotiation successful
0807 18:31:14.101787 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":370,"threads_started":1}
I20260807 18:31:14.104977 10507 heartbeater.cc:347] Connected to a master server at 127.27.1.254:33219
I20260807 18:31:14.105293 10507 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:14.106022 10507 heartbeater.cc:511] Master 127.27.1.254:33219 requested a full tablet report, sending...
I20260807 18:31:14.107535 10167 ts_manager.cc:194] Registered new tserver with Master: 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141)
I20260807 18:31:14.108357 10167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:48507
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:31:14.149894 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:33219
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:14.488955 10513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:14.489281 10513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:14.489435 10513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:14.497516 10513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:14.497664 10513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:14.497879 10513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:31:14.507879 10513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:14.509143 10513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:14.510574 10513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:14.524502 10518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:14.524513 10521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:14.524515 10519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:14.525729 10513 server_base.cc:1034] running on GCE node
I20260807 18:31:14.526360 10513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:14.527283 10513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:14.528488 10513 hybrid_clock.cc:648] HybridClock initialized: now 1786127474528434 us; error 54 us; skew 500 ppm
I20260807 18:31:14.528748 10513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:14 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127474, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:14.531843 10513 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:31:14.533192 10513 webserver.cc:460] Webserver started at http://127.27.1.195:44247/ using document root <none> and password file <none>
I20260807 18:31:14.533818 10513 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:14.533921 10513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:14.534349 10513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:14.537497 10513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "3bc7d47aab6f46a8bb48efc2d813dade"
format_stamp: "Formatted at 2026-08-07 18:31:14 on dist-test-slave-1xrh"
server_key: "b911c320f24ec008babf64a139c68dd6"
server_key_iv: "2d702331168b661523b5e916276ddec2"
server_key_version: "encryptionkey@0"
I20260807 18:31:14.538365 10513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "3bc7d47aab6f46a8bb48efc2d813dade"
format_stamp: "Formatted at 2026-08-07 18:31:14 on dist-test-slave-1xrh"
server_key: "b911c320f24ec008babf64a139c68dd6"
server_key_iv: "2d702331168b661523b5e916276ddec2"
server_key_version: "encryptionkey@0"
I20260807 18:31:14.543704 10513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 18:31:14.547195 10529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:14.548499 10513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:31:14.548678 10513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "3bc7d47aab6f46a8bb48efc2d813dade"
format_stamp: "Formatted at 2026-08-07 18:31:14 on dist-test-slave-1xrh"
server_key: "b911c320f24ec008babf64a139c68dd6"
server_key_iv: "2d702331168b661523b5e916276ddec2"
server_key_version: "encryptionkey@0"
I20260807 18:31:14.548883 10513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:14.569063 10513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:14.572945 10513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:14.573221 10513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:14.574142 10513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:14.575351 10366 heartbeater.cc:503] Master 127.27.1.254:33219 was elected leader, sending a full tablet report...
I20260807 18:31:14.575590 10513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:14.575683 10513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:14.575817 10513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:14.575891 10513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:14.609881 10513 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:42675
I20260807 18:31:14.609929 10647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:42675 every 8 connection(s)
I20260807 18:31:14.611191 10513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:31:14.617547 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 10513
I20260807 18:31:14.617758 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:31:14.618216 27655 external_mini_cluster.cc:1428] Setting key 933be90ad864ea2290954e8b13eca7fc
Aug 07 18:31:14 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127474, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:14.631754 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.613683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:59283 (local address 127.27.1.254:33219)
0807 18:31:14.614257 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:14.614261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:14.615026 (+ 765us) server_negotiation.cc:407] Connection header received
0807 18:31:14.617039 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:14.617045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:14.617169 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:14.617301 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:14.619111 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.620405 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.622182 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.622581 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.622711 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.626134 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:14.626140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:14.626145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:14.626187 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:14.628695 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:14.629412 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:14.629419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:14.629423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:14.629555 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:14.629990 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:14.629995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:14.629999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:14.630292 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:14.630464 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:14.631103 (+ 639us) server_negotiation.cc:299] Negotiation successful
0807 18:31:14.631351 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":307,"threads_started":1}
I20260807 18:31:14.632900 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.614173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:59283)
0807 18:31:14.614846 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:14.614900 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:14.616761 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:14.617496 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:14.617504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:14.618204 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:14.618934 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.618964 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.620568 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.620572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.622023 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.622037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.622835 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.622843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.623081 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.623745 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:14.623767 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:14.625917 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:14.628837 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:14.628846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:14.628862 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:14.629284 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:14.629672 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:14.629677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:14.629680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:14.629871 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:14.630433 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:14.630441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:14.630754 (+ 313us) client_negotiation.cc:769] Sending connection context
0807 18:31:14.631023 (+ 269us) client_negotiation.cc:240] Negotiation successful
0807 18:31:14.631569 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":252,"threads_started":1}
I20260807 18:31:14.634378 10648 heartbeater.cc:347] Connected to a master server at 127.27.1.254:33219
I20260807 18:31:14.634642 10648 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:14.635419 10648 heartbeater.cc:511] Master 127.27.1.254:33219 requested a full tablet report, sending...
I20260807 18:31:14.636750 10167 ts_manager.cc:194] Registered new tserver with Master: 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
I20260807 18:31:14.637507 10167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:59283
I20260807 18:31:14.646000 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:31:14.663455 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.650577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.27.1.254:33219)
0807 18:31:14.650768 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:14.650776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:14.651192 (+ 416us) server_negotiation.cc:407] Connection header received
0807 18:31:14.651401 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:14.651408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:14.651479 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:14.651583 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:14.653038 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.653870 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.655182 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.655603 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.655740 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.658008 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:14.658013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:14.658016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:14.658047 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:14.660445 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:14.661427 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:14.661433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:14.661437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:14.661570 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:14.661961 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:14.661967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:14.661971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:14.662289 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:14.662548 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:14.662865 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 18:31:14.663106 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 18:31:14.667373 10167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:31:14.674226 10167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:31:14.710166 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.686452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:42675 (local address 127.0.0.1:55536)
0807 18:31:14.700636 (+ 14184us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:14.700664 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:14.700805 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:14.702264 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:14.702270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:14.702300 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:14.702781 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.702795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.705324 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.705329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.707821 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.707834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.709070 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.709077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.709266 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.709318 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 18:31:14.709484 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 18:31:14.709680 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14025,"mutex_wait_us":118,"thread_start_us":15592,"threads_started":1}
I20260807 18:31:14.710780 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.687507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.27.1.193:44151)
0807 18:31:14.701809 (+ 14302us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:14.701818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:14.702489 (+ 671us) server_negotiation.cc:407] Connection header received
0807 18:31:14.702832 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:14.702840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:14.703050 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:14.703273 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:14.703862 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.704877 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.707915 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.709317 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.709447 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.709640 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:14.709955 (+ 315us) server_negotiation.cc:299] Negotiation successful
0807 18:31:14.710124 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14133,"thread_start_us":13981,"threads_started":1}
I20260807 18:31:14.710947 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.686220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:34141 (local address 127.0.0.1:48870)
0807 18:31:14.699387 (+ 13167us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:14.699443 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:14.699657 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:14.701865 (+ 2208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:14.701879 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:14.701923 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:14.703273 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.703285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.705394 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.705398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.707564 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.707577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.709732 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.709737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.709909 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.710267 (+ 358us) client_negotiation.cc:769] Sending connection context
0807 18:31:14.710357 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 18:31:14.710492 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12848,"mutex_wait_us":27,"thread_start_us":14086,"threads_started":1}
I20260807 18:31:14.711076 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.687597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.27.1.195:42675)
0807 18:31:14.701448 (+ 13851us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:14.701457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:14.701494 (+ 37us) server_negotiation.cc:407] Connection header received
0807 18:31:14.701594 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:14.701602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:14.701809 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:14.702412 (+ 603us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:14.703534 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.705195 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.707992 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.708883 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.709045 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.709275 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:14.710521 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0807 18:31:14.710686 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13681,"thread_start_us":13515,"threads_started":1}
I20260807 18:31:14.711459 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.686811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44151 (local address 127.0.0.1:52510)
0807 18:31:14.702361 (+ 15550us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:14.702387 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:14.702599 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:14.703271 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:14.703275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:14.703299 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:14.703687 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.703696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.705017 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.705022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.707743 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.707759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.709530 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.709536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.709724 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.709788 (+ 64us) client_negotiation.cc:769] Sending connection context
0807 18:31:14.709887 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 18:31:14.710009 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15417,"mutex_wait_us":60,"thread_start_us":12134,"threads_started":1}
I20260807 18:31:14.711755 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.686867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.27.1.194:34141)
0807 18:31:14.697247 (+ 10380us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:14.697260 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:14.699891 (+ 2631us) server_negotiation.cc:407] Connection header received
0807 18:31:14.700015 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:14.700026 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:14.700316 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:14.700520 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:14.703466 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.705187 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.707719 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.709200 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.709360 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.709564 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:14.711233 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0807 18:31:14.711414 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10158,"thread_start_us":10034,"threads_started":1}
I20260807 18:31:14.714751 10294 tablet_service.cc:1467] Processing CreateTablet for tablet c655cec775f0496a853033459fb09ba1 (DEFAULT_TABLE table=test-table [id=9e7d95962c954d7f95486e750bb49e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:14.715015 10578 tablet_service.cc:1467] Processing CreateTablet for tablet c655cec775f0496a853033459fb09ba1 (DEFAULT_TABLE table=test-table [id=9e7d95962c954d7f95486e750bb49e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:14.715317 10439 tablet_service.cc:1467] Processing CreateTablet for tablet c655cec775f0496a853033459fb09ba1 (DEFAULT_TABLE table=test-table [id=9e7d95962c954d7f95486e750bb49e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:14.716604 10294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c655cec775f0496a853033459fb09ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:14.716647 10578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c655cec775f0496a853033459fb09ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:14.716666 10439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c655cec775f0496a853033459fb09ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:14.728873 10669 tablet_bootstrap.cc:492] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: Bootstrap starting.
I20260807 18:31:14.731477 10670 tablet_bootstrap.cc:492] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: Bootstrap starting.
I20260807 18:31:14.732482 10669 tablet_bootstrap.cc:654] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:14.733825 10669 log.cc:824] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:14.735169 10670 tablet_bootstrap.cc:654] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:14.736466 10670 log.cc:824] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:14.737381 10669 tablet_bootstrap.cc:492] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: No bootstrap required, opened a new log
I20260807 18:31:14.737653 10669 ts_tablet_manager.cc:1397] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260807 18:31:14.735380 10671 tablet_bootstrap.cc:492] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: Bootstrap starting.
I20260807 18:31:14.739431 10670 tablet_bootstrap.cc:492] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: No bootstrap required, opened a new log
I20260807 18:31:14.739702 10670 ts_tablet_manager.cc:1397] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260807 18:31:14.742717 10671 tablet_bootstrap.cc:654] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:14.744006 10671 log.cc:824] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:14.745122 10669 raft_consensus.cc:348] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } }
I20260807 18:31:14.745604 10669 raft_consensus.cc:374] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:14.745724 10669 raft_consensus.cc:729] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4299a843b4049ceaa60048e64bebc80, State: Initialized, Role: FOLLOWER
I20260807 18:31:14.746336 10669 consensus_queue.cc:260] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } }
I20260807 18:31:14.747197 10671 tablet_bootstrap.cc:492] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: No bootstrap required, opened a new log
I20260807 18:31:14.747718 10671 ts_tablet_manager.cc:1397] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260807 18:31:14.748104 10669 ts_tablet_manager.cc:1428] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260807 18:31:14.750257 10670 raft_consensus.cc:348] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } }
I20260807 18:31:14.750697 10670 raft_consensus.cc:374] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:14.750799 10670 raft_consensus.cc:729] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc7d47aab6f46a8bb48efc2d813dade, State: Initialized, Role: FOLLOWER
I20260807 18:31:14.751546 10670 consensus_queue.cc:260] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } }
I20260807 18:31:14.753412 10648 heartbeater.cc:503] Master 127.27.1.254:33219 was elected leader, sending a full tablet report...
I20260807 18:31:14.754000 10670 ts_tablet_manager.cc:1428] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260807 18:31:14.757493 10671 raft_consensus.cc:348] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } }
I20260807 18:31:14.757962 10671 raft_consensus.cc:374] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:14.758100 10671 raft_consensus.cc:729] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841c868a39524c218efc67bbc5e46a53, State: Initialized, Role: FOLLOWER
I20260807 18:31:14.758665 10671 consensus_queue.cc:260] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } }
I20260807 18:31:14.759826 10507 heartbeater.cc:503] Master 127.27.1.254:33219 was elected leader, sending a full tablet report...
I20260807 18:31:14.760494 10671 ts_tablet_manager.cc:1428] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
W20260807 18:31:14.793100 10367 tablet.cc:2369] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:31:14.830124 10508 tablet.cc:2369] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:14.830999 10676 raft_consensus.cc:482] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:31:14.831327 10676 raft_consensus.cc:504] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } }
I20260807 18:31:14.833010 10676 leader_election.cc:288] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141), 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
I20260807 18:31:14.839948 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.833527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:57053 (local address 127.27.1.195:42675)
0807 18:31:14.833762 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:14.833770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:14.834546 (+ 776us) server_negotiation.cc:407] Connection header received
0807 18:31:14.834760 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:14.834764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:14.834823 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:14.834919 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:14.835662 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.836395 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.838239 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.838991 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.839196 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.839276 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:14.839559 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 18:31:14.839681 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 18:31:14.840047 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.833355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:40943 (local address 127.27.1.194:34141)
0807 18:31:14.833588 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:14.833596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:14.834475 (+ 879us) server_negotiation.cc:407] Connection header received
0807 18:31:14.834732 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:14.834738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:14.834841 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:14.834952 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:14.835670 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.836383 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.838289 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.839059 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.839282 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.839363 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:14.839654 (+ 291us) server_negotiation.cc:299] Negotiation successful
0807 18:31:14.839779 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:31:14.840070 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.833461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:42675 (local address 127.27.1.193:57053)
0807 18:31:14.834455 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:14.834469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:14.834597 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:14.834937 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:14.834941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:14.834973 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:14.835506 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.835515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.836533 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.836541 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.838081 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.838097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.839236 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.839242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.839399 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.839429 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 18:31:14.839535 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 18:31:14.839657 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":84,"thread_start_us":443,"threads_started":1}
I20260807 18:31:14.840174 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.833461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:34141 (local address 127.27.1.193:40943)
0807 18:31:14.834362 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:14.834378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:14.834561 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:14.834969 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:14.834974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:14.834997 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:14.835517 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.835527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.836547 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.836553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.838124 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:14.838137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.839332 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.839339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:14.839495 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.839517 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 18:31:14.839625 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 18:31:14.839745 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":58,"thread_start_us":518,"threads_started":1}
I20260807 18:31:14.841190 10462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c655cec775f0496a853033459fb09ba1" candidate_uuid: "d4299a843b4049ceaa60048e64bebc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841c868a39524c218efc67bbc5e46a53" is_pre_election: true
I20260807 18:31:14.841174 10599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c655cec775f0496a853033459fb09ba1" candidate_uuid: "d4299a843b4049ceaa60048e64bebc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" is_pre_election: true
I20260807 18:31:14.841667 10462 raft_consensus.cc:2454] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4299a843b4049ceaa60048e64bebc80 in term 0.
I20260807 18:31:14.841701 10599 raft_consensus.cc:2454] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4299a843b4049ceaa60048e64bebc80 in term 0.
I20260807 18:31:14.842327 10248 leader_election.cc:302] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841c868a39524c218efc67bbc5e46a53, d4299a843b4049ceaa60048e64bebc80; no voters:
I20260807 18:31:14.842746 10676 raft_consensus.cc:2781] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:31:14.842902 10676 raft_consensus.cc:482] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:31:14.843019 10676 raft_consensus.cc:3037] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:14.845180 10676 raft_consensus.cc:504] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } }
I20260807 18:31:14.845781 10676 leader_election.cc:288] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 1 election: Requested vote from peers 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141), 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
I20260807 18:31:14.846190 10462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c655cec775f0496a853033459fb09ba1" candidate_uuid: "d4299a843b4049ceaa60048e64bebc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841c868a39524c218efc67bbc5e46a53"
I20260807 18:31:14.846294 10599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c655cec775f0496a853033459fb09ba1" candidate_uuid: "d4299a843b4049ceaa60048e64bebc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc7d47aab6f46a8bb48efc2d813dade"
I20260807 18:31:14.846403 10462 raft_consensus.cc:3037] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:14.846500 10599 raft_consensus.cc:3037] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:14.849692 10462 raft_consensus.cc:2454] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4299a843b4049ceaa60048e64bebc80 in term 1.
I20260807 18:31:14.849692 10599 raft_consensus.cc:2454] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4299a843b4049ceaa60048e64bebc80 in term 1.
I20260807 18:31:14.850097 10250 leader_election.cc:302] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc7d47aab6f46a8bb48efc2d813dade, d4299a843b4049ceaa60048e64bebc80; no voters:
I20260807 18:31:14.850364 10676 raft_consensus.cc:2781] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:31:14.850915 10676 raft_consensus.cc:686] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 1 LEADER]: Becoming Leader. State: Replica: d4299a843b4049ceaa60048e64bebc80, State: Running, Role: LEADER
I20260807 18:31:14.851503 10676 consensus_queue.cc:237] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } }
I20260807 18:31:14.856417 10166 catalog_manager.cc:5495] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4299a843b4049ceaa60048e64bebc80 (127.27.1.193). New cstate: current_term: 1 leader_uuid: "d4299a843b4049ceaa60048e64bebc80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } health_report { overall_health: HEALTHY } } }
W20260807 18:31:14.862788 10649 tablet.cc:2369] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:14.901015 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.896027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.27.1.193:44151)
0807 18:31:14.896206 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:14.896211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:14.896229 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:31:14.896342 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:14.896346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:14.896436 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:14.896559 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:31:14.896890 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.897581 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.898974 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.899450 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.899544 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.900067 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:31:14.900553 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:31:14.900659 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:14.900744 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 18:31:14.900807 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 18:31:14.908865 10462 raft_consensus.cc:1261] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53 [term 1 FOLLOWER]: Refusing update from remote peer d4299a843b4049ceaa60048e64bebc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:31:14.908888 10599 raft_consensus.cc:1261] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade [term 1 FOLLOWER]: Refusing update from remote peer d4299a843b4049ceaa60048e64bebc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:31:14.909832 10688 consensus_queue.cc:1035] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:31:14.910161 10676 consensus_queue.cc:1035] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:31:14.922490 10693 mvcc.cc:205] Tried to move back new op lower bound from 7315978137216950272 to 7315978136996200448. Current Snapshot: MvccSnapshot[applied={T|T < 7315978137216950272}]
I20260807 18:31:14.934926 10166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:31:14.935300 10166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:31:14.938814 10166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Sending DeleteTablet for 3 replicas of tablet c655cec775f0496a853033459fb09ba1
I20260807 18:31:14.939985 10439 tablet_service.cc:1514] Processing DeleteTablet for tablet c655cec775f0496a853033459fb09ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:14 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:48870
I20260807 18:31:14.940155 10578 tablet_service.cc:1514] Processing DeleteTablet for tablet c655cec775f0496a853033459fb09ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:14 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:55536
I20260807 18:31:14.940330 10294 tablet_service.cc:1514] Processing DeleteTablet for tablet c655cec775f0496a853033459fb09ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:14 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:52510
I20260807 18:31:14.941390 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 10229
I20260807 18:31:14.952589 10701 tablet_replica.cc:330] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: stopping tablet replica
I20260807 18:31:14.953198 10701 raft_consensus.cc:2229] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:31:14.953961 10701 raft_consensus.cc:2258] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:14.956678 10701 ts_tablet_manager.cc:1905] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:31:14.965256 10701 ts_tablet_manager.cc:1918] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:31:14.965505 10701 log.cc:1196] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/c655cec775f0496a853033459fb09ba1
I20260807 18:31:14.966198 10701 ts_tablet_manager.cc:1939] T c655cec775f0496a853033459fb09ba1 P 841c868a39524c218efc67bbc5e46a53: Deleting consensus metadata
I20260807 18:31:14.968281 10148 catalog_manager.cc:4841] TS 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141): tablet c655cec775f0496a853033459fb09ba1 (table test-table [id=9e7d95962c954d7f95486e750bb49e68]) successfully deleted
I20260807 18:31:14.971622 10702 tablet_replica.cc:330] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: stopping tablet replica
I20260807 18:31:14.971633 10704 tablet_replica.cc:330] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: stopping tablet replica
I20260807 18:31:14.972124 10704 raft_consensus.cc:2229] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:31:14.972159 10702 raft_consensus.cc:2229] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:31:14.972652 10702 raft_consensus.cc:2258] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:14.973039 10704 raft_consensus.cc:2258] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:14.974917 10702 ts_tablet_manager.cc:1905] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:31:14.975383 10704 ts_tablet_manager.cc:1905] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 18:31:14 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:31:14.983011 10702 ts_tablet_manager.cc:1918] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:31:14.983254 10702 log.cc:1196] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/c655cec775f0496a853033459fb09ba1
I20260807 18:31:14.983669 10704 ts_tablet_manager.cc:1918] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:31:14.983915 10704 log.cc:1196] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/c655cec775f0496a853033459fb09ba1
I20260807 18:31:14.983933 10702 ts_tablet_manager.cc:1939] T c655cec775f0496a853033459fb09ba1 P 3bc7d47aab6f46a8bb48efc2d813dade: Deleting consensus metadata
I20260807 18:31:14.984589 10704 ts_tablet_manager.cc:1939] T c655cec775f0496a853033459fb09ba1 P d4299a843b4049ceaa60048e64bebc80: Deleting consensus metadata
I20260807 18:31:14.985947 10151 catalog_manager.cc:4841] TS 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675): tablet c655cec775f0496a853033459fb09ba1 (table test-table [id=9e7d95962c954d7f95486e750bb49e68]) successfully deleted
I20260807 18:31:14.986356 10147 catalog_manager.cc:4841] TS d4299a843b4049ceaa60048e64bebc80 (127.27.1.193:44151): tablet c655cec775f0496a853033459fb09ba1 (table test-table [id=9e7d95962c954d7f95486e750bb49e68]) successfully deleted
I20260807 18:31:14.986826 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:14.942817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.27.1.193:44151)
0807 18:31:14.943061 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:14.943071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:14.964928 (+ 21857us) server_negotiation.cc:407] Connection header received
0807 18:31:14.965505 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:14.965510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:14.965614 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:14.965700 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:14.967857 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.969042 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.970429 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:14.971384 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:14.971532 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:14.978068 (+ 6536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:14.978100 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:14.978117 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:14.978171 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:14.982830 (+ 4659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:14.984048 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:14.984056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:14.984059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:14.984248 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:14.984769 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:14.984778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:14.984784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:14.985231 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:14.985567 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:14.986187 (+ 620us) server_negotiation.cc:299] Negotiation successful
0807 18:31:14.986438 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:31:14.988080 10360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:15.096251 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 10229
I20260807 18:31:15.116390 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 10372
Aug 07 18:31:15 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:31:15.136715 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:15.117275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.27.1.194:34141)
0807 18:31:15.117537 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:15.117542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:15.117563 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:31:15.117704 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:15.117710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:15.117850 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:15.117976 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:15.119328 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:15.120095 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:15.121514 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:15.121900 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:15.122039 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:15.130869 (+ 8830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:15.130883 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:15.130899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:15.130937 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:15.134140 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:15.134848 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:15.134855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:15.134858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:15.134984 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:15.135374 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:15.135377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:15.135379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:15.135752 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:15.135976 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:15.136261 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 18:31:15.136429 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:31:15.137444 10502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:15.228063 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 10372
I20260807 18:31:15.249420 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 10513
Aug 07 18:31:15 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:31:15.265511 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:15.250192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.27.1.195:42675)
0807 18:31:15.250363 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:15.250367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:15.250386 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:31:15.250533 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:15.250537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:15.250608 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:15.250694 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:15.252067 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:15.252910 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:15.254401 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:15.254962 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:15.255060 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:15.259707 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:15.259722 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:15.259738 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:15.259776 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:15.262510 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:15.263435 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:15.263443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:15.263446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:15.263591 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:15.263992 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:15.263996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:15.263998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:15.264330 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:15.264645 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:15.264970 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 18:31:15.265187 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 18:31:15.266132 10643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:15.358233 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 10513
I20260807 18:31:15.378700 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 10131
I20260807 18:31:15.380065 10197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:15.504153 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 10131
I20260807 18:31:15.531721 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:33219
--webserver_interface=127.27.1.254
--webserver_port=46787
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:33219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:15.869338 10712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:15.869671 10712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:15.869853 10712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:15.876976 10712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:31:15.877113 10712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:15.877198 10712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:15.877286 10712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:31:15.877351 10712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:31:15.887135 10712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:33219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:33219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=46787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:15.888471 10712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:15.889814 10712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:15.916939 10718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:15.916944 10719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:15.917409 10721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:15.917886 10712 server_base.cc:1034] running on GCE node
I20260807 18:31:15.918823 10712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:15.920536 10712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:15.921782 10712 hybrid_clock.cc:648] HybridClock initialized: now 1786127475921725 us; error 59 us; skew 500 ppm
I20260807 18:31:15.922253 10712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:15 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127475, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:15.925868 10712 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:31:15.927654 10712 webserver.cc:460] Webserver started at http://127.27.1.254:46787/ using document root <none> and password file <none>
I20260807 18:31:15.928385 10712 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:15.928512 10712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:15.934443 10712 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 18:31:15.937327 10730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:15.938673 10712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:31:15.938860 10712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45"
format_stamp: "Formatted at 2026-08-07 18:31:12 on dist-test-slave-1xrh"
server_key: "8aede755105c6f73773c4d05622f9835"
server_key_iv: "bea453d3e7aa58eb8a015774011c983e"
server_key_version: "encryptionkey@0"
I20260807 18:31:15.939983 10712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:15.962483 10712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:15.966578 10712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:15.966811 10712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:15.986188 10712 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:33219
I20260807 18:31:15.986208 10788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:33219 every 8 connection(s)
I20260807 18:31:15.987881 10712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:31:15.988947 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 10712
I20260807 18:31:15.989804 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:44151
--local_ip_for_outbound_sockets=127.27.1.193
--tserver_master_addrs=127.27.1.254:33219
--webserver_port=37701
--webserver_interface=127.27.1.193
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:31:15.995195 10789 sys_catalog.cc:263] Verifying existing consensus state
I20260807 18:31:15.998514 10789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Bootstrap starting.
I20260807 18:31:16.018078 10789 log.cc:824] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:16.030229 10789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 18:31:16.030798 10789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Bootstrap complete.
I20260807 18:31:16.039836 10789 raft_consensus.cc:348] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:16.040673 10789 raft_consensus.cc:729] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffdcd7fda3943d8a7ec70e1c4d26d45, State: Initialized, Role: FOLLOWER
I20260807 18:31:16.041239 10789 consensus_queue.cc:260] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:16.041498 10789 raft_consensus.cc:388] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:31:16.041626 10789 raft_consensus.cc:482] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:31:16.041800 10789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 1 FOLLOWER]: Advancing to term 2
I20260807 18:31:16.045800 10789 raft_consensus.cc:504] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:16.046298 10789 leader_election.cc:302] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bffdcd7fda3943d8a7ec70e1c4d26d45; no voters:
I20260807 18:31:16.046937 10789 leader_election.cc:288] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 18:31:16.047078 10793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 18:31:16.049450 10789 sys_catalog.cc:564] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:31:16.054289 10793 raft_consensus.cc:686] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 2 LEADER]: Becoming Leader. State: Replica: bffdcd7fda3943d8a7ec70e1c4d26d45, State: Running, Role: LEADER
I20260807 18:31:16.054944 10793 consensus_queue.cc:237] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:16.058646 10794 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } } }
I20260807 18:31:16.059048 10794 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:16.059456 10795 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bffdcd7fda3943d8a7ec70e1c4d26d45. Latest consensus state: current_term: 2 leader_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } } }
I20260807 18:31:16.059790 10795 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:16.065728 10808 catalog_manager.cc:1259] Loaded cluster ID: 2a3feffc53304f7faf53facd6d05de3d
I20260807 18:31:16.065891 10808 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: loading cluster ID for follower catalog manager: success
W20260807 18:31:16.068629 10808 catalog_manager.cc:1576] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260807 18:31:16.068827 10808 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260807 18:31:16.069464 10809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:31:16.073019 10809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:31:16.073875 10809 catalog_manager.cc:1259] Loaded cluster ID: 2a3feffc53304f7faf53facd6d05de3d
I20260807 18:31:16.073992 10809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:31:16.075884 10809 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 18:31:16.086539 10809 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 18:31:16.093370 10809 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:31:16.094619 10809 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Loaded TSK: 0
I20260807 18:31:16.095813 10809 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 18:31:16.363940 10791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:16.364264 10791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:16.364431 10791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:16.371944 10791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:16.372088 10791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:16.372251 10791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:31:16.382058 10791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:44151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=37701
--tserver_master_addrs=127.27.1.254:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:16.383844 10791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:16.385176 10791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:16.400916 10821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:16.403487 10818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:16.400801 10819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:16.404699 10791 server_base.cc:1034] running on GCE node
I20260807 18:31:16.405304 10791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:16.406334 10791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:16.407541 10791 hybrid_clock.cc:648] HybridClock initialized: now 1786127476407497 us; error 38 us; skew 500 ppm
I20260807 18:31:16.407850 10791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:16 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127476, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:16.411245 10791 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:31:16.412801 10791 webserver.cc:460] Webserver started at http://127.27.1.193:37701/ using document root <none> and password file <none>
I20260807 18:31:16.413483 10791 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:16.413641 10791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:16.419027 10791 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:31:16.421957 10828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:16.423410 10791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:31:16.423595 10791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "d4299a843b4049ceaa60048e64bebc80"
format_stamp: "Formatted at 2026-08-07 18:31:13 on dist-test-slave-1xrh"
server_key: "255a46d1e022f89df9cf3600ca79d5d6"
server_key_iv: "d9c8fd1d5b59f95e0064889fc647e497"
server_key_version: "encryptionkey@0"
I20260807 18:31:16.424563 10791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:16.451778 10791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:16.455904 10791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:16.456192 10791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:16.457211 10791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:16.458516 10791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:16.458629 10791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:16.458762 10791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:16.458850 10791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:16.492118 10791 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:44151
I20260807 18:31:16.492204 10944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:44151 every 8 connection(s)
I20260807 18:31:16.493978 10791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:31:16.498870 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 10791
I20260807 18:31:16.499637 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:34141
--local_ip_for_outbound_sockets=127.27.1.194
--tserver_master_addrs=127.27.1.254:33219
--webserver_port=38507
--webserver_interface=127.27.1.194
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:31:16 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127476, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:16.521467 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:16.497343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:44381)
0807 18:31:16.499667 (+ 2324us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:16.499713 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:16.501402 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:16.502779 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:16.502793 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:16.503627 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:16.504447 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:16.504474 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:16.506087 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:16.506095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:16.508246 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:16.508265 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:16.509231 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:16.509240 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:16.509558 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:16.510322 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:16.510345 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:16.512687 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:16.517338 (+ 4651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:16.517345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:16.517356 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:16.517762 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:16.518230 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:16.518234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:16.518236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:16.518401 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:16.519433 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:16.519440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:16.519790 (+ 350us) client_negotiation.cc:769] Sending connection context
0807 18:31:16.520106 (+ 316us) client_negotiation.cc:240] Negotiation successful
0807 18:31:16.520484 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2025,"thread_start_us":1816,"threads_started":1}
I20260807 18:31:16.522051 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:16.497148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:44381 (local address 127.27.1.254:33219)
0807 18:31:16.498013 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:16.498031 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:16.501694 (+ 3663us) server_negotiation.cc:407] Connection header received
0807 18:31:16.501965 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:16.501975 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:16.502266 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:16.502550 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:16.504663 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:16.505911 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:16.508389 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:16.508997 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:16.509126 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:16.512973 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:16.512986 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:16.513007 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:16.513064 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:16.517204 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:16.517912 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:16.517923 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:16.517932 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:16.518120 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:16.518520 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:16.518527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:16.518533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:16.519259 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:16.519463 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:16.520400 (+ 937us) server_negotiation.cc:299] Negotiation successful
0807 18:31:16.520855 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":401,"threads_started":1}
I20260807 18:31:16.524611 10946 heartbeater.cc:347] Connected to a master server at 127.27.1.254:33219
I20260807 18:31:16.524921 10946 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:16.525633 10946 heartbeater.cc:511] Master 127.27.1.254:33219 requested a full tablet report, sending...
I20260807 18:31:16.527722 10749 ts_manager.cc:194] Registered new tserver with Master: d4299a843b4049ceaa60048e64bebc80 (127.27.1.193:44151)
I20260807 18:31:16.529189 10749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:44381
W20260807 18:31:16.854302 10950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:16.854619 10950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:16.854779 10950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:16.861927 10950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:16.862067 10950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:16.862246 10950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:31:16.871693 10950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:34141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=38507
--tserver_master_addrs=127.27.1.254:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:16.872956 10950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:16.874270 10950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:16.887422 10956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:16.887512 10959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:16.887454 10957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:16.888243 10950 server_base.cc:1034] running on GCE node
I20260807 18:31:16.888826 10950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:16.889698 10950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:16.890913 10950 hybrid_clock.cc:648] HybridClock initialized: now 1786127476890873 us; error 33 us; skew 500 ppm
I20260807 18:31:16.891268 10950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:16 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127476, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:16.894263 10950 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:31:16.895766 10950 webserver.cc:460] Webserver started at http://127.27.1.194:38507/ using document root <none> and password file <none>
I20260807 18:31:16.896423 10950 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:16.896571 10950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:16.901626 10950 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 18:31:16.904196 10966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:16.905498 10950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:31:16.905670 10950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "841c868a39524c218efc67bbc5e46a53"
format_stamp: "Formatted at 2026-08-07 18:31:13 on dist-test-slave-1xrh"
server_key: "e13dd5afc85e33d14a7d3289deea9ca6"
server_key_iv: "aadd88ec397d409715ea30109c7385b8"
server_key_version: "encryptionkey@0"
I20260807 18:31:16.906636 10950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:16.926430 10950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:16.930418 10950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:16.930689 10950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:16.931825 10950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:16.933118 10950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:16.933239 10950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:16.933372 10950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:16.933457 10950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:16.965534 10950 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:34141
I20260807 18:31:16.965626 11090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:34141 every 8 connection(s)
I20260807 18:31:16.966958 10950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:31:16.972330 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 10950
I20260807 18:31:16.972939 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:42675
--local_ip_for_outbound_sockets=127.27.1.195
--tserver_master_addrs=127.27.1.254:33219
--webserver_port=44247
--webserver_interface=127.27.1.195
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:31:16 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127476, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:16.989553 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:16.969706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:38947 (local address 127.27.1.254:33219)
0807 18:31:16.969885 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:16.969890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:16.971017 (+ 1127us) server_negotiation.cc:407] Connection header received
0807 18:31:16.972984 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:16.972990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:16.973098 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:16.973223 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:16.975333 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:16.976045 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:16.978560 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:16.979003 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:16.979098 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:16.983408 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:16.983417 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:16.983425 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:16.983472 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:16.986285 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:16.987092 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:16.987102 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:16.987105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:16.987276 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:16.987794 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:16.987798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:16.987800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:16.988098 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:16.988258 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:16.988960 (+ 702us) server_negotiation.cc:299] Negotiation successful
0807 18:31:16.989211 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:31:16.990593 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:16.970178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:38947)
0807 18:31:16.970865 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:16.970906 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:16.972694 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:16.973472 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:16.973484 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:16.974290 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:16.975154 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:16.975183 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:16.976220 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:16.976228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:16.978384 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:16.978403 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:16.979268 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:16.979278 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:16.979615 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:16.980396 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:16.980436 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:16.983113 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:16.986434 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:16.986443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:16.986460 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:16.986959 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:16.987390 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:16.987395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:16.987398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:16.987593 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:16.988231 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:16.988239 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:16.988585 (+ 346us) client_negotiation.cc:769] Sending connection context
0807 18:31:16.988848 (+ 263us) client_negotiation.cc:240] Negotiation successful
0807 18:31:16.989376 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":264,"threads_started":1}
I20260807 18:31:16.992166 11092 heartbeater.cc:347] Connected to a master server at 127.27.1.254:33219
I20260807 18:31:16.992524 11092 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:16.993319 11092 heartbeater.cc:511] Master 127.27.1.254:33219 requested a full tablet report, sending...
I20260807 18:31:16.994788 10749 ts_manager.cc:194] Registered new tserver with Master: 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141)
I20260807 18:31:16.995723 10749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:38947
W20260807 18:31:17.318977 11095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:17.319339 11095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:17.319497 11095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:17.327478 11095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:17.327615 11095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:17.327805 11095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:31:17.337850 11095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:42675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=44247
--tserver_master_addrs=127.27.1.254:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:17.339347 11095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:17.340775 11095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:17.354724 11102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:17.355234 11104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:17.355370 11101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:17.356047 11095 server_base.cc:1034] running on GCE node
I20260807 18:31:17.356660 11095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:17.357618 11095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:17.358922 11095 hybrid_clock.cc:648] HybridClock initialized: now 1786127477358865 us; error 40 us; skew 500 ppm
I20260807 18:31:17.359341 11095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:17 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127477, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:17.362533 11095 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:31:17.364138 11095 webserver.cc:460] Webserver started at http://127.27.1.195:44247/ using document root <none> and password file <none>
I20260807 18:31:17.364876 11095 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:17.365034 11095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:17.370405 11095 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260807 18:31:17.373263 11112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:17.374651 11095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:31:17.374907 11095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "3bc7d47aab6f46a8bb48efc2d813dade"
format_stamp: "Formatted at 2026-08-07 18:31:14 on dist-test-slave-1xrh"
server_key: "b911c320f24ec008babf64a139c68dd6"
server_key_iv: "2d702331168b661523b5e916276ddec2"
server_key_version: "encryptionkey@0"
I20260807 18:31:17.376034 11095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:17.406714 11095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:17.410769 11095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:17.411051 11095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:17.412125 11095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:17.413444 11095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:17.413564 11095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:17.413697 11095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:17.413781 11095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:17.447557 11095 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:42675
I20260807 18:31:17.447642 11235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:42675 every 8 connection(s)
I20260807 18:31:17.449168 11095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:31:17.456605 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 11095
Aug 07 18:31:17 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127477, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:17.470898 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.451881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:54589 (local address 127.27.1.254:33219)
0807 18:31:17.452061 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:17.452066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:17.453139 (+ 1073us) server_negotiation.cc:407] Connection header received
0807 18:31:17.455091 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:17.455097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:17.455211 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:17.455321 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:17.457132 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.458337 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.459974 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.460436 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.460608 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.464172 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:17.464178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:17.464181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:17.464212 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:17.467289 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:17.468253 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:17.468257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:17.468259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:17.468416 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:17.469075 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:17.469079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:17.469081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:17.469389 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:17.469585 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:17.470305 (+ 720us) server_negotiation.cc:299] Negotiation successful
0807 18:31:17.470545 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":58}
I20260807 18:31:17.472033 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.457598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.27.1.254:33219)
0807 18:31:17.458133 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:17.458139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:17.458413 (+ 274us) server_negotiation.cc:407] Connection header received
0807 18:31:17.458602 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:17.458611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:17.458666 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:17.458774 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:17.460379 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.461183 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.462461 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.462891 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.463033 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.465778 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:17.465783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:17.465787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:17.465827 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:17.469075 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:17.469758 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:17.469765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:17.469769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:17.469894 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:17.470270 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:17.470276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:17.470279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:17.470546 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:17.470809 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:17.471398 (+ 589us) server_negotiation.cc:299] Negotiation successful
0807 18:31:17.471650 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":265,"threads_started":1}
I20260807 18:31:17.472064 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.452297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:54589)
0807 18:31:17.452982 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:17.453029 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:17.454785 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:17.455557 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:17.455566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:17.456067 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:17.456905 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.456933 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.458507 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.458517 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.459809 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.459842 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.460701 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.460710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.460961 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.461639 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:17.461660 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:17.463959 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:17.467439 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:17.467446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:17.467460 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:17.468132 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:17.468588 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:17.468597 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:17.468602 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:17.468869 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:17.469556 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:17.469565 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:17.469916 (+ 351us) client_negotiation.cc:769] Sending connection context
0807 18:31:17.470219 (+ 303us) client_negotiation.cc:240] Negotiation successful
0807 18:31:17.470766 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":262,"threads_started":1}
I20260807 18:31:17.473615 11236 heartbeater.cc:347] Connected to a master server at 127.27.1.254:33219
I20260807 18:31:17.473917 11236 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:17.474736 11236 heartbeater.cc:511] Master 127.27.1.254:33219 requested a full tablet report, sending...
I20260807 18:31:17.476128 10749 ts_manager.cc:194] Registered new tserver with Master: 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
I20260807 18:31:17.476922 10749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:54589
I20260807 18:31:17.486725 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:31:17.493731 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.487757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.27.1.254:33219)
0807 18:31:17.487914 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:17.487918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:17.488335 (+ 417us) server_negotiation.cc:407] Connection header received
0807 18:31:17.488523 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:17.488526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:17.488651 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:17.488769 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:31:17.489150 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.489821 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.491277 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.491918 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.492052 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.492550 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:31:17.493219 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:31:17.493349 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:17.493400 (+ 51us) server_negotiation.cc:299] Negotiation successful
0807 18:31:17.493477 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 18:31:17.494638 10749 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:31:17.503057 10749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:31:17.526703 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.514537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:34141 (local address 127.0.0.1:48882)
0807 18:31:17.515372 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:17.515415 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:17.515564 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:17.519849 (+ 4285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:17.519854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:17.519881 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:17.520329 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.520349 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.521882 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.521893 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.524070 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.524083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.525597 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.525605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.525874 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.525945 (+ 71us) client_negotiation.cc:769] Sending connection context
0807 18:31:17.526048 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 18:31:17.526287 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":94,"thread_start_us":414,"threads_started":1}
I20260807 18:31:17.526847 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.516072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.27.1.193:44151)
0807 18:31:17.516963 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:17.516972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:17.517605 (+ 633us) server_negotiation.cc:407] Connection header received
0807 18:31:17.517713 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:17.517719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:17.517907 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:17.518105 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:17.519264 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.520755 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.524107 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.525404 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.525561 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.525814 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:17.526126 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 18:31:17.526297 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":541,"threads_started":1}
I20260807 18:31:17.527004 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.514706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.27.1.194:34141)
0807 18:31:17.519240 (+ 4534us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:17.519247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:17.519270 (+ 23us) server_negotiation.cc:407] Connection header received
0807 18:31:17.519387 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:17.519397 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:17.519660 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:17.519867 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:17.520572 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.521682 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.524222 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.525330 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.526216 (+ 886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.526407 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:17.526488 (+ 81us) server_negotiation.cc:299] Negotiation successful
0807 18:31:17.526649 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4348,"thread_start_us":4223,"threads_started":1}
I20260807 18:31:17.528093 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.514893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44151 (local address 127.0.0.1:52532)
0807 18:31:17.517303 (+ 2410us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:17.517327 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:17.517469 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:17.518116 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:17.518127 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:17.518158 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:17.518972 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.518985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.520968 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.520974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.523916 (+ 2942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.523937 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.525642 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.525649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.525901 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.525947 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 18:31:17.526080 (+ 133us) client_negotiation.cc:240] Negotiation successful
0807 18:31:17.527683 (+ 1603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2260,"mutex_wait_us":291,"thread_start_us":3342,"threads_started":1}
I20260807 18:31:17.528532 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.516807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:42675 (local address 127.0.0.1:55556)
0807 18:31:17.517451 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:17.517467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:17.517560 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:17.520152 (+ 2592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:17.520159 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:17.520182 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:17.520674 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.520687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.522687 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.522693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.524870 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.524887 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.527366 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.527372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.527571 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.527680 (+ 109us) client_negotiation.cc:769] Sending connection context
0807 18:31:17.527747 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 18:31:17.527838 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":24,"thread_start_us":1213,"threads_started":1}
I20260807 18:31:17.529220 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.517616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.27.1.195:42675)
0807 18:31:17.518816 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:17.518827 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:17.519401 (+ 574us) server_negotiation.cc:407] Connection header received
0807 18:31:17.519516 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:17.519525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:17.519749 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:17.519941 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:17.520881 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.522520 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.525050 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.526619 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.528163 (+ 1544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.528571 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:17.528775 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 18:31:17.528895 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":833,"threads_started":1}
I20260807 18:31:17.531159 11020 tablet_service.cc:1467] Processing CreateTablet for tablet b38ff64ab9544e1bbe808397520b07ee (DEFAULT_TABLE table=test-table [id=98ff6370c0e34eb4a6ff30bc4251efcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:17.532114 10946 heartbeater.cc:503] Master 127.27.1.254:33219 was elected leader, sending a full tablet report...
I20260807 18:31:17.532737 11020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38ff64ab9544e1bbe808397520b07ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:17.532596 11167 tablet_service.cc:1467] Processing CreateTablet for tablet b38ff64ab9544e1bbe808397520b07ee (DEFAULT_TABLE table=test-table [id=98ff6370c0e34eb4a6ff30bc4251efcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:17.533967 11167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38ff64ab9544e1bbe808397520b07ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:17.534976 10878 tablet_service.cc:1467] Processing CreateTablet for tablet b38ff64ab9544e1bbe808397520b07ee (DEFAULT_TABLE table=test-table [id=98ff6370c0e34eb4a6ff30bc4251efcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:17.536033 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38ff64ab9544e1bbe808397520b07ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:17.548599 11255 tablet_bootstrap.cc:492] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Bootstrap starting.
I20260807 18:31:17.551605 11257 tablet_bootstrap.cc:492] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: Bootstrap starting.
I20260807 18:31:17.552569 11255 tablet_bootstrap.cc:654] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:17.553987 11255 log.cc:824] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:17.554908 11256 tablet_bootstrap.cc:492] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: Bootstrap starting.
I20260807 18:31:17.555579 11257 tablet_bootstrap.cc:654] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:17.557094 11257 log.cc:824] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:17.557686 11256 tablet_bootstrap.cc:654] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:17.559649 11256 log.cc:824] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:17.560420 11257 tablet_bootstrap.cc:492] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: No bootstrap required, opened a new log
I20260807 18:31:17.560711 11257 ts_tablet_manager.cc:1397] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260807 18:31:17.561795 11255 tablet_bootstrap.cc:492] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: No bootstrap required, opened a new log
I20260807 18:31:17.562096 11255 ts_tablet_manager.cc:1397] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260807 18:31:17.562657 11256 tablet_bootstrap.cc:492] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: No bootstrap required, opened a new log
I20260807 18:31:17.562935 11256 ts_tablet_manager.cc:1397] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 18:31:17.569146 11257 raft_consensus.cc:348] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:17.569522 11257 raft_consensus.cc:374] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:17.569630 11257 raft_consensus.cc:729] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841c868a39524c218efc67bbc5e46a53, State: Initialized, Role: FOLLOWER
I20260807 18:31:17.570248 11257 consensus_queue.cc:260] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:17.571359 11092 heartbeater.cc:503] Master 127.27.1.254:33219 was elected leader, sending a full tablet report...
I20260807 18:31:17.573244 11257 ts_tablet_manager.cc:1428] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260807 18:31:17.573141 11255 raft_consensus.cc:348] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:17.573611 11255 raft_consensus.cc:374] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:17.573716 11255 raft_consensus.cc:729] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4299a843b4049ceaa60048e64bebc80, State: Initialized, Role: FOLLOWER
I20260807 18:31:17.574326 11255 consensus_queue.cc:260] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:17.574363 11256 raft_consensus.cc:348] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:17.574805 11256 raft_consensus.cc:374] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:17.574998 11256 raft_consensus.cc:729] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc7d47aab6f46a8bb48efc2d813dade, State: Initialized, Role: FOLLOWER
I20260807 18:31:17.575769 11256 consensus_queue.cc:260] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:17.576154 11255 ts_tablet_manager.cc:1428] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260807 18:31:17.578413 11256 ts_tablet_manager.cc:1428] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260807 18:31:17.578510 11236 heartbeater.cc:503] Master 127.27.1.254:33219 was elected leader, sending a full tablet report...
W20260807 18:31:17.700899 11237 tablet.cc:2369] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:31:17.718978 11093 tablet.cc:2369] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:31:17.746372 10947 tablet.cc:2369] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:17.799268 11262 raft_consensus.cc:482] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:31:17.799582 11262 raft_consensus.cc:504] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:17.801311 11262 leader_election.cc:288] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141), 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
I20260807 18:31:17.808584 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.801881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:46919 (local address 127.27.1.195:42675)
0807 18:31:17.802100 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:17.802105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:17.803016 (+ 911us) server_negotiation.cc:407] Connection header received
0807 18:31:17.803219 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:17.803223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:17.803287 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:17.803387 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:17.804030 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.804785 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.806687 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.807532 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.807716 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.807788 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:17.808163 (+ 375us) server_negotiation.cc:299] Negotiation successful
0807 18:31:17.808323 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 18:31:17.808625 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.801939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:42675 (local address 127.27.1.193:46919)
0807 18:31:17.802876 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:17.802892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:17.803034 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:17.803403 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:17.803408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:17.803432 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:17.803855 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.803867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.804930 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.804936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.806532 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.806545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.807733 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.807738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.807916 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.807950 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 18:31:17.808111 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 18:31:17.808239 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":33,"thread_start_us":602,"threads_started":1}
I20260807 18:31:17.808703 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.801797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:53061 (local address 127.27.1.194:34141)
0807 18:31:17.802033 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:17.802041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:17.803099 (+ 1058us) server_negotiation.cc:407] Connection header received
0807 18:31:17.803323 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:17.803329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:17.803400 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:17.803497 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:17.804162 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.804869 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.806689 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.807548 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.807712 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.807788 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:17.808329 (+ 541us) server_negotiation.cc:299] Negotiation successful
0807 18:31:17.808464 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 18:31:17.809381 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.801953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:34141 (local address 127.27.1.193:53061)
0807 18:31:17.802980 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:17.803001 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:17.803177 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:17.803520 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:17.803524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:17.803563 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:17.803990 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.804000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.805011 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.805021 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.806534 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:17.806546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.807958 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.807977 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:17.808145 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.808177 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 18:31:17.808890 (+ 713us) client_negotiation.cc:240] Negotiation successful
0807 18:31:17.809021 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":65,"thread_start_us":529,"threads_started":1}
I20260807 18:31:17.810309 11187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38ff64ab9544e1bbe808397520b07ee" candidate_uuid: "d4299a843b4049ceaa60048e64bebc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" is_pre_election: true
I20260807 18:31:17.810345 11042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38ff64ab9544e1bbe808397520b07ee" candidate_uuid: "d4299a843b4049ceaa60048e64bebc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841c868a39524c218efc67bbc5e46a53" is_pre_election: true
I20260807 18:31:17.810935 11042 raft_consensus.cc:2454] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4299a843b4049ceaa60048e64bebc80 in term 0.
I20260807 18:31:17.810940 11187 raft_consensus.cc:2454] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4299a843b4049ceaa60048e64bebc80 in term 0.
I20260807 18:31:17.811636 10830 leader_election.cc:302] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841c868a39524c218efc67bbc5e46a53, d4299a843b4049ceaa60048e64bebc80; no voters:
I20260807 18:31:17.812024 11262 raft_consensus.cc:2781] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:31:17.812197 11262 raft_consensus.cc:482] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:31:17.812312 11262 raft_consensus.cc:3037] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:17.814533 11262 raft_consensus.cc:504] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:17.815155 11262 leader_election.cc:288] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 1 election: Requested vote from peers 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141), 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
I20260807 18:31:17.815517 11042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38ff64ab9544e1bbe808397520b07ee" candidate_uuid: "d4299a843b4049ceaa60048e64bebc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841c868a39524c218efc67bbc5e46a53"
I20260807 18:31:17.815665 11187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38ff64ab9544e1bbe808397520b07ee" candidate_uuid: "d4299a843b4049ceaa60048e64bebc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc7d47aab6f46a8bb48efc2d813dade"
I20260807 18:31:17.815732 11042 raft_consensus.cc:3037] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:17.815874 11187 raft_consensus.cc:3037] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:17.818987 11187 raft_consensus.cc:2454] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4299a843b4049ceaa60048e64bebc80 in term 1.
I20260807 18:31:17.818997 11042 raft_consensus.cc:2454] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4299a843b4049ceaa60048e64bebc80 in term 1.
I20260807 18:31:17.819424 10830 leader_election.cc:302] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841c868a39524c218efc67bbc5e46a53, d4299a843b4049ceaa60048e64bebc80; no voters:
I20260807 18:31:17.819713 11262 raft_consensus.cc:2781] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:31:17.820190 11262 raft_consensus.cc:686] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 LEADER]: Becoming Leader. State: Replica: d4299a843b4049ceaa60048e64bebc80, State: Running, Role: LEADER
I20260807 18:31:17.820657 11262 consensus_queue.cc:237] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:17.825098 10749 catalog_manager.cc:5495] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4299a843b4049ceaa60048e64bebc80 (127.27.1.193). New cstate: current_term: 1 leader_uuid: "d4299a843b4049ceaa60048e64bebc80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } health_report { overall_health: UNKNOWN } } }
I20260807 18:31:17.891324 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.886327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.27.1.193:44151)
0807 18:31:17.886532 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:17.886536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:17.886572 (+ 36us) server_negotiation.cc:407] Connection header received
0807 18:31:17.886678 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:17.886682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:17.886758 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:17.886892 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:31:17.887299 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.888098 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.889414 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.889805 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.889893 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.890326 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:31:17.890825 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:31:17.890937 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:17.891010 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 18:31:17.891076 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 18:31:17.899302 11187 raft_consensus.cc:1261] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade [term 1 FOLLOWER]: Refusing update from remote peer d4299a843b4049ceaa60048e64bebc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:31:17.899305 11042 raft_consensus.cc:1261] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53 [term 1 FOLLOWER]: Refusing update from remote peer d4299a843b4049ceaa60048e64bebc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:31:17.900404 11262 consensus_queue.cc:1035] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:31:17.900736 11266 consensus_queue.cc:1035] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:31:17.915613 11270 mvcc.cc:205] Tried to move back new op lower bound from 7315978149464616960 to 7315978149156839424. Current Snapshot: MvccSnapshot[applied={T|T < 7315978149464616960}]
I20260807 18:31:17.925418 10748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:31:17.925678 10748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:31:17.929347 10748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Sending DeleteTablet for 3 replicas of tablet b38ff64ab9544e1bbe808397520b07ee
I20260807 18:31:17.930588 11020 tablet_service.cc:1514] Processing DeleteTablet for tablet b38ff64ab9544e1bbe808397520b07ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:17 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:48882
I20260807 18:31:17.930773 11167 tablet_service.cc:1514] Processing DeleteTablet for tablet b38ff64ab9544e1bbe808397520b07ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:17 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:55556
I20260807 18:31:17.931358 10878 tablet_service.cc:1514] Processing DeleteTablet for tablet b38ff64ab9544e1bbe808397520b07ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:17 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:52532
I20260807 18:31:17.931713 11279 tablet_replica.cc:330] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: stopping tablet replica
I20260807 18:31:17.932163 11279 raft_consensus.cc:2229] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:31:17.932250 11278 tablet_replica.cc:330] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: stopping tablet replica
I20260807 18:31:17.932461 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 10791
I20260807 18:31:17.932675 11278 raft_consensus.cc:2229] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:31:17.932758 11279 raft_consensus.cc:2258] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:17.933127 11278 raft_consensus.cc:2258] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:17.935305 11278 ts_tablet_manager.cc:1905] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:31:17.937682 11279 ts_tablet_manager.cc:1905] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:31:17.942302 11278 ts_tablet_manager.cc:1918] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:31:17.942541 11278 log.cc:1196] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/b38ff64ab9544e1bbe808397520b07ee
I20260807 18:31:17.943212 11278 ts_tablet_manager.cc:1939] T b38ff64ab9544e1bbe808397520b07ee P 841c868a39524c218efc67bbc5e46a53: Deleting consensus metadata
I20260807 18:31:17.945292 10732 catalog_manager.cc:4841] TS 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141): tablet b38ff64ab9544e1bbe808397520b07ee (table test-table [id=98ff6370c0e34eb4a6ff30bc4251efcf]) successfully deleted
I20260807 18:31:17.950671 11280 tablet_replica.cc:330] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: stopping tablet replica
I20260807 18:31:17.951238 11280 raft_consensus.cc:2229] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:31:17.951575 11279 ts_tablet_manager.cc:1918] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:31:17.951784 11279 log.cc:1196] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/b38ff64ab9544e1bbe808397520b07ee
I20260807 18:31:17.952342 11279 ts_tablet_manager.cc:1939] T b38ff64ab9544e1bbe808397520b07ee P 3bc7d47aab6f46a8bb48efc2d813dade: Deleting consensus metadata
I20260807 18:31:17.952291 11280 raft_consensus.cc:2258] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:17.954211 10734 catalog_manager.cc:4841] TS 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675): tablet b38ff64ab9544e1bbe808397520b07ee (table test-table [id=98ff6370c0e34eb4a6ff30bc4251efcf]) successfully deleted
I20260807 18:31:17.954631 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:17.933325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.27.1.193:44151)
0807 18:31:17.933492 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:17.933498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:17.933549 (+ 51us) server_negotiation.cc:407] Connection header received
0807 18:31:17.933759 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:17.933763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:17.933852 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:17.933961 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:17.936518 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.937755 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.940387 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:17.941083 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:17.941212 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:17.945520 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:17.945535 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:17.945552 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:17.945607 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:17.950358 (+ 4751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:17.951392 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:17.951400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:17.951404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:17.951589 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:17.952132 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:17.952143 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:17.952146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:17.952638 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:17.953202 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:17.953835 (+ 633us) server_negotiation.cc:299] Negotiation successful
0807 18:31:17.954265 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 18:31:17.954630 11280 ts_tablet_manager.cc:1905] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:31:17.955870 10940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:18.054200 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 10791
I20260807 18:31:18.075785 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 10950
W20260807 18:31:18.075788 10731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.1.193:44151 (error 108)
I20260807 18:31:18.077831 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:18.077094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44151 (local address 127.0.0.1:52562)
0807 18:31:18.077563 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:18.077666 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:44151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":202,"threads_started":1}
W20260807 18:31:18.078087 10731 catalog_manager.cc:4584] TS d4299a843b4049ceaa60048e64bebc80 (127.27.1.193:44151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b38ff64ab9544e1bbe808397520b07ee: Network error: Client connection negotiation failed: client connection to 127.27.1.193:44151: connect: Connection refused (error 111)
I20260807 18:31:18.090185 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:18.076809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.27.1.194:34141)
0807 18:31:18.077011 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:18.077019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:18.077039 (+ 20us) server_negotiation.cc:407] Connection header received
0807 18:31:18.077104 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:18.077109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:18.077211 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:18.077330 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:18.078879 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:18.079712 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:18.081096 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:18.081498 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:18.081630 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:18.084599 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:18.084612 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:18.084627 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:18.084659 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:18.087330 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:18.088016 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:18.088022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:18.088025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:18.088187 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:18.088595 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:18.088600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:18.088604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:18.088979 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:18.089271 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:18.089600 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 18:31:18.089828 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 18:31:18.090837 11086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:18.131560 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:18.131064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44151 (local address 127.0.0.1:52570)
0807 18:31:18.131301 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:18.131398 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:44151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 18:31:18.132630 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:18.132261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44151 (local address 127.0.0.1:52572)
0807 18:31:18.132441 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:18.132503 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:44151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 18:31:18.179998 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 10950
I20260807 18:31:18.193325 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:18.192833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44151 (local address 127.0.0.1:52582)
0807 18:31:18.193078 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:18.193173 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:44151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 18:31:18.194416 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:18.194055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44151 (local address 127.0.0.1:52588)
0807 18:31:18.194205 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:18.194285 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:44151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 18:31:18.201443 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 11095
I20260807 18:31:18.214439 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:18.202219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.27.1.195:42675)
0807 18:31:18.202438 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:18.202443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:18.202462 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:31:18.202613 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:18.202616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:18.202682 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:18.202767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:18.204199 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:18.204907 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:18.206196 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:18.206580 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:18.206670 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:18.209149 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:18.209164 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:18.209181 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:18.209220 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:18.212056 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:18.212728 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:18.212734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:18.212736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:18.212869 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:18.213169 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:18.213172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:18.213173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:18.213512 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:18.213716 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:18.213918 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 18:31:18.214079 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 18:31:18.215073 11230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:18.275265 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:18.274633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44151 (local address 127.0.0.1:52604)
0807 18:31:18.274894 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:18.275010 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:44151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 18:31:18.276405 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:18.275938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44151 (local address 127.0.0.1:52614)
0807 18:31:18.276162 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:18.276249 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:44151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260807 18:31:18.304642 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 11095
I20260807 18:31:18.324802 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 10712
I20260807 18:31:18.325982 10783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:18.441452 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 10712
I20260807 18:31:18.467207 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:33219
--webserver_interface=127.27.1.254
--webserver_port=46787
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:33219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:18.806529 11287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:18.806857 11287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:18.807005 11287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:18.814208 11287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:31:18.814342 11287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:18.814465 11287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:18.814559 11287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:31:18.814642 11287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:31:18.824404 11287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:33219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:33219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=46787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:18.825686 11287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:18.826977 11287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:18.838877 11294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:18.838888 11293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:18.838784 11296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:18.839771 11287 server_base.cc:1034] running on GCE node
I20260807 18:31:18.840523 11287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:18.841862 11287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:18.843056 11287 hybrid_clock.cc:648] HybridClock initialized: now 1786127478843000 us; error 48 us; skew 500 ppm
I20260807 18:31:18.843421 11287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:18 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127478, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:18.846474 11287 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:31:18.847980 11287 webserver.cc:460] Webserver started at http://127.27.1.254:46787/ using document root <none> and password file <none>
I20260807 18:31:18.848703 11287 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:18.848842 11287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:18.853982 11287 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 18:31:18.856715 11304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:18.857946 11287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 18:31:18.858119 11287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45"
format_stamp: "Formatted at 2026-08-07 18:31:12 on dist-test-slave-1xrh"
server_key: "8aede755105c6f73773c4d05622f9835"
server_key_iv: "bea453d3e7aa58eb8a015774011c983e"
server_key_version: "encryptionkey@0"
I20260807 18:31:18.859072 11287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:18.876690 11287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:18.880434 11287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:18.880692 11287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:18.898907 11287 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:33219
I20260807 18:31:18.898960 11358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:33219 every 8 connection(s)
I20260807 18:31:18.900329 11287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:31:18.903033 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 11287
I20260807 18:31:18.904093 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:44151
--local_ip_for_outbound_sockets=127.27.1.193
--tserver_master_addrs=127.27.1.254:33219
--webserver_port=37701
--webserver_interface=127.27.1.193
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:31:18.909271 11359 sys_catalog.cc:263] Verifying existing consensus state
I20260807 18:31:18.914363 11359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Bootstrap starting.
I20260807 18:31:18.935225 11359 log.cc:824] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:18.951717 11359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 18:31:18.952282 11359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45: Bootstrap complete.
I20260807 18:31:18.961462 11359 raft_consensus.cc:348] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:18.962471 11359 raft_consensus.cc:729] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffdcd7fda3943d8a7ec70e1c4d26d45, State: Initialized, Role: FOLLOWER
I20260807 18:31:18.963222 11359 consensus_queue.cc:260] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:18.963476 11359 raft_consensus.cc:388] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:31:18.963601 11359 raft_consensus.cc:482] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:31:18.963806 11359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 2 FOLLOWER]: Advancing to term 3
I20260807 18:31:18.967658 11359 raft_consensus.cc:504] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:18.968185 11359 leader_election.cc:302] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bffdcd7fda3943d8a7ec70e1c4d26d45; no voters:
I20260807 18:31:18.968870 11359 leader_election.cc:288] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 18:31:18.969015 11364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 18:31:18.970284 11364 raft_consensus.cc:686] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [term 3 LEADER]: Becoming Leader. State: Replica: bffdcd7fda3943d8a7ec70e1c4d26d45, State: Running, Role: LEADER
I20260807 18:31:18.970924 11364 consensus_queue.cc:237] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } }
I20260807 18:31:18.971159 11359 sys_catalog.cc:564] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:31:18.973587 11366 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bffdcd7fda3943d8a7ec70e1c4d26d45. Latest consensus state: current_term: 3 leader_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } } }
I20260807 18:31:18.973624 11365 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffdcd7fda3943d8a7ec70e1c4d26d45" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33219 } } }
I20260807 18:31:18.973974 11366 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:18.974054 11365 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffdcd7fda3943d8a7ec70e1c4d26d45 [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:18.984745 11372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:31:18.994160 11372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:31:18.995705 11372 catalog_manager.cc:1259] Loaded cluster ID: 2a3feffc53304f7faf53facd6d05de3d
I20260807 18:31:18.995889 11372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 18:31:18.998114 11372 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f388272f9ad google::LogMessage::Fail() at ??:0
@ 0x7f388273288a google::LogMessage::SendToLog() at ??:0
@ 0x7f388272f4de google::LogMessage::Flush() at ??:0
@ 0x7f3882730a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3898162512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f389815f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f38981e7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f38981e7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f388537b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f388537e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f388537e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3885359eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f38886336db start_thread at ??:0
@ 0x7f38815c961f clone at ??:0
W20260807 18:31:19.286278 11362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:19.286655 11362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:19.286835 11362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:19.294034 11362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:19.294176 11362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:19.294332 11362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:31:19.303941 11362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:44151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=37701
--tserver_master_addrs=127.27.1.254:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:19.305212 11362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:19.306517 11362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:19.322669 11405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:19.322737 11406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:19.322737 11408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:19.323599 11362 server_base.cc:1034] running on GCE node
I20260807 18:31:19.324216 11362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:19.325166 11362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:19.326416 11362 hybrid_clock.cc:648] HybridClock initialized: now 1786127479326356 us; error 69 us; skew 500 ppm
I20260807 18:31:19.326792 11362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:19 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:19.337059 11362 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:31:19.338598 11362 webserver.cc:460] Webserver started at http://127.27.1.193:37701/ using document root <none> and password file <none>
I20260807 18:31:19.339349 11362 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:19.339506 11362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:19.344762 11362 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260807 18:31:19.347847 11415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:19.349224 11362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:31:19.349361 11362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "d4299a843b4049ceaa60048e64bebc80"
format_stamp: "Formatted at 2026-08-07 18:31:13 on dist-test-slave-1xrh"
server_key: "255a46d1e022f89df9cf3600ca79d5d6"
server_key_iv: "d9c8fd1d5b59f95e0064889fc647e497"
server_key_version: "encryptionkey@0"
I20260807 18:31:19.350279 11362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:19.366916 11362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:19.371240 11362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:19.371477 11362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:19.372421 11362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:19.374172 11424 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 18:31:19.379865 11362 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 18:31:19.379988 11362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260807 18:31:19.380085 11362 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 18:31:19.383095 11362 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 18:31:19.383224 11362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260807 18:31:19.383610 11424 tablet_bootstrap.cc:492] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Bootstrap starting.
I20260807 18:31:19.392299 11424 log.cc:824] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:19.408679 11424 tablet_bootstrap.cc:492] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 18:31:19.409693 11424 tablet_bootstrap.cc:492] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Bootstrap complete.
I20260807 18:31:19.410413 11424 ts_tablet_manager.cc:1397] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.005s
I20260807 18:31:19.422432 11424 raft_consensus.cc:348] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:19.423800 11424 raft_consensus.cc:729] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4299a843b4049ceaa60048e64bebc80, State: Initialized, Role: FOLLOWER
I20260807 18:31:19.424503 11424 consensus_queue.cc:260] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:19.426206 11424 ts_tablet_manager.cc:1428] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260807 18:31:19.430477 11362 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:44151
I20260807 18:31:19.430562 11540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:44151 every 8 connection(s)
I20260807 18:31:19.431739 11362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:31:19.435105 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 11362
I20260807 18:31:19.435786 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:34141
--local_ip_for_outbound_sockets=127.27.1.194
--tserver_master_addrs=127.27.1.254:33219
--webserver_port=38507
--webserver_interface=127.27.1.194
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:31:19.436697 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:19.434514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:35589)
0807 18:31:19.435411 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:19.435707 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":375,"threads_started":1}
W20260807 18:31:19.437299 11541 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:33219 (0 consecutive failures): Network error: Failed to ping master at 127.27.1.254:33219: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
I20260807 18:31:19.438198 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:19.437892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:41703)
0807 18:31:19.438032 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:19.438099 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 18:31:19.439163 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:19.438815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:49779)
0807 18:31:19.438938 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:19.439006 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260807 18:31:19.439414 11541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 18:31:19.772765 11544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:19.773039 11544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:19.773236 11544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:19.780629 11544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:19.780731 11544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:19.780897 11544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:31:19.790490 11544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:34141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=38507
--tserver_master_addrs=127.27.1.254:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:19.791872 11544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:19.793213 11544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:19.806439 11549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:19.806401 11550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:19.806475 11552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:19.807430 11544 server_base.cc:1034] running on GCE node
I20260807 18:31:19.808091 11544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:19.809082 11544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:19.810330 11544 hybrid_clock.cc:648] HybridClock initialized: now 1786127479810285 us; error 34 us; skew 500 ppm
I20260807 18:31:19.810660 11544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:19 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:19.814083 11544 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:31:19.815625 11544 webserver.cc:460] Webserver started at http://127.27.1.194:38507/ using document root <none> and password file <none>
I20260807 18:31:19.816332 11544 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:19.816481 11544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:19.821666 11544 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 18:31:19.824437 11559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:19.825799 11544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:31:19.825989 11544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "841c868a39524c218efc67bbc5e46a53"
format_stamp: "Formatted at 2026-08-07 18:31:13 on dist-test-slave-1xrh"
server_key: "e13dd5afc85e33d14a7d3289deea9ca6"
server_key_iv: "aadd88ec397d409715ea30109c7385b8"
server_key_version: "encryptionkey@0"
I20260807 18:31:19.827003 11544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:19.848277 11544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:19.852615 11544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:19.852932 11544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:19.854045 11544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:19.855515 11544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:19.855640 11544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:19.855770 11544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:19.855870 11544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:19.888839 11544 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:34141
I20260807 18:31:19.888923 11677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:34141 every 8 connection(s)
I20260807 18:31:19.890337 11544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:31:19.894927 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 11544
I20260807 18:31:19.895301 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:19.893299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:49035)
0807 18:31:19.894042 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:19.894247 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":270,"threads_started":1}
I20260807 18:31:19.895735 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:42675
--local_ip_for_outbound_sockets=127.27.1.195
--tserver_master_addrs=127.27.1.254:33219
--webserver_port=44247
--webserver_interface=127.27.1.195
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:19.896037 11678 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:33219 (0 consecutive failures): Network error: Failed to ping master at 127.27.1.254:33219: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
I20260807 18:31:19.896898 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:19.896588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:56479)
0807 18:31:19.896731 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:19.896797 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 18:31:19.897780 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:19.897494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:59087)
0807 18:31:19.897615 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:19.897682 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 18:31:19.898036 11678 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 18:31:20.230571 11681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:20.230863 11681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:20.230986 11681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:20.238632 11681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:20.238744 11681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:20.238865 11681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:31:20.248812 11681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:42675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=44247
--tserver_master_addrs=127.27.1.254:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:20.250182 11681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:20.251462 11681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:20.264617 11686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:20.264559 11689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:20.264756 11687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:20.265779 11681 server_base.cc:1034] running on GCE node
I20260807 18:31:20.266348 11681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:20.267382 11681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:20.268584 11681 hybrid_clock.cc:648] HybridClock initialized: now 1786127480268524 us; error 50 us; skew 500 ppm
I20260807 18:31:20.268852 11681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:20 dist-test-slave-1xrh krb5kdc[10118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:20.272063 11681 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:31:20.273442 11681 webserver.cc:460] Webserver started at http://127.27.1.195:44247/ using document root <none> and password file <none>
I20260807 18:31:20.274102 11681 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:20.274237 11681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:20.279417 11681 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260807 18:31:20.281922 11697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:20.283288 11681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:31:20.283459 11681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "3bc7d47aab6f46a8bb48efc2d813dade"
format_stamp: "Formatted at 2026-08-07 18:31:14 on dist-test-slave-1xrh"
server_key: "b911c320f24ec008babf64a139c68dd6"
server_key_iv: "2d702331168b661523b5e916276ddec2"
server_key_version: "encryptionkey@0"
I20260807 18:31:20.284426 11681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:20.302642 11681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:20.306535 11681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:20.306803 11681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:20.307826 11681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:20.309098 11681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:20.309216 11681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:20.309345 11681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:20.309429 11681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:20.343400 11681 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:42675
I20260807 18:31:20.343497 11815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:42675 every 8 connection(s)
I20260807 18:31:20.344889 11681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:31:20.349673 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:20.347825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:52321)
0807 18:31:20.348611 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:20.348817 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":292,"threads_started":1}
W20260807 18:31:20.350317 11816 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:33219 (0 consecutive failures): Network error: Failed to ping master at 127.27.1.254:33219: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
I20260807 18:31:20.351094 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:20.350784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:58567)
0807 18:31:20.350917 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:20.350986 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 18:31:20.351907 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:20.351615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:36067)
0807 18:31:20.351729 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:20.351799 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260807 18:31:20.352126 11816 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 18:31:20.353914 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 11681
I20260807 18:31:20.440682 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:20.439901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:42451)
0807 18:31:20.440361 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:20.440480 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":224,"threads_started":1}
I20260807 18:31:20.754148 11821 raft_consensus.cc:482] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:31:20.754487 11821 raft_consensus.cc:504] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:20.756173 11821 leader_election.cc:288] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141), 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
Aug 07 18:31:20 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
Aug 07 18:31:20 dist-test-slave-1xrh krb5kdc[10118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:31:20.782687 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:20.756821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:54081 (local address 127.27.1.195:42675)
0807 18:31:20.759265 (+ 2444us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:20.759278 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:20.759318 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:31:20.761416 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:20.761438 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:20.761802 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:20.762120 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:20.764092 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:20.765967 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:20.768768 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:20.769791 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:20.770002 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:20.774560 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:20.774569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:20.774582 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:20.774656 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:20.778546 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:20.779749 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:20.779758 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:20.779767 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:20.779975 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:20.780641 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:20.780647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:20.780650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:20.781346 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:20.781546 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:20.782079 (+ 533us) server_negotiation.cc:299] Negotiation successful
0807 18:31:20.782396 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2239,"thread_start_us":2061,"threads_started":1}
I20260807 18:31:20.782918 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:20.756537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:34141 (local address 127.27.1.193:57433)
0807 18:31:20.758509 (+ 1972us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:20.758531 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:20.758680 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:20.762701 (+ 4021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:20.762704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:20.763272 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:20.763837 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:20.763857 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:20.766189 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:20.766193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:20.768618 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:20.768634 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:20.770056 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:20.770063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:20.770466 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:20.771561 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:20.771584 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:20.774529 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:20.778717 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:20.778727 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:20.778743 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:20.779564 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:20.780137 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:20.780141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:20.780144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:20.780352 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:20.781200 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:20.781208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:20.781550 (+ 342us) client_negotiation.cc:769] Sending connection context
0807 18:31:20.781882 (+ 332us) client_negotiation.cc:240] Negotiation successful
0807 18:31:20.782447 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1804,"mutex_wait_us":535,"spinlock_wait_cycles":49536,"thread_start_us":1313,"threads_started":1}
I20260807 18:31:20.783140 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:20.756532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:42675 (local address 127.27.1.193:54081)
0807 18:31:20.756753 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:20.756791 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:20.757502 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:20.762299 (+ 4797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:20.762310 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:20.763065 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:20.763834 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:20.763854 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:20.766184 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:20.766190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:20.768617 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:20.768634 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:20.770046 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:20.770053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:20.770463 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:20.771576 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:20.771597 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:20.774268 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:20.778717 (+ 4449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:20.778727 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:20.778743 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:20.779564 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:20.780142 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:20.780150 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:20.780156 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:20.780496 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:20.781521 (+ 1025us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:20.781526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:20.781691 (+ 165us) client_negotiation.cc:769] Sending connection context
0807 18:31:20.781882 (+ 191us) client_negotiation.cc:240] Negotiation successful
0807 18:31:20.782572 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":587,"spinlock_wait_cycles":2176}
I20260807 18:31:20.784111 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:20.756945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:57433 (local address 127.27.1.194:34141)
0807 18:31:20.758840 (+ 1895us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:20.758857 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:20.758904 (+ 47us) server_negotiation.cc:407] Connection header received
0807 18:31:20.761176 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:20.761191 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:20.761722 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:20.762120 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:20.764045 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:20.765967 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:20.768768 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:20.769791 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:20.770002 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:20.774795 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:20.774804 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:20.774818 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:20.774872 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:20.778579 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:20.779748 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:20.779758 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:20.779767 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:20.779993 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:20.780487 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:20.780495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:20.780499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:20.781058 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:20.781263 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:20.783245 (+ 1982us) server_negotiation.cc:299] Negotiation successful
0807 18:31:20.783660 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1688,"thread_start_us":1566,"threads_started":1}
W20260807 18:31:20.786432 11420 leader_election.cc:341] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675): Not found: Tablet not found: b38ff64ab9544e1bbe808397520b07ee
W20260807 18:31:20.786733 11417 leader_election.cc:341] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141): Not found: Tablet not found: b38ff64ab9544e1bbe808397520b07ee
I20260807 18:31:20.786872 11417 leader_election.cc:302] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4299a843b4049ceaa60048e64bebc80; no voters: 3bc7d47aab6f46a8bb48efc2d813dade, 841c868a39524c218efc67bbc5e46a53
I20260807 18:31:20.787288 11821 raft_consensus.cc:2726] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:31:20.899545 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:20.898640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:39605)
0807 18:31:20.899097 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:20.899304 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":220,"threads_started":1}
I20260807 18:31:21.353479 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:21.352618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:52097)
0807 18:31:21.353141 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:21.353267 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":295,"threads_started":1}
I20260807 18:31:21.442397 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:21.441608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:32809)
0807 18:31:21.442037 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:21.442160 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":213,"threads_started":1}
I20260807 18:31:21.901037 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:21.900374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:60017)
0807 18:31:21.900784 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:21.900863 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":215,"threads_started":1}
I20260807 18:31:22.355057 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:22.354368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:57287)
0807 18:31:22.354790 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:22.354874 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":214,"threads_started":1}
I20260807 18:31:22.444186 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:22.443441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:58239)
0807 18:31:22.443916 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:22.444011 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":227,"threads_started":1}
I20260807 18:31:22.743644 11831 raft_consensus.cc:482] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:31:22.743906 11831 raft_consensus.cc:504] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:22.744593 11831 leader_election.cc:288] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141), 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
W20260807 18:31:22.745630 11417 leader_election.cc:341] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141): Not found: Tablet not found: b38ff64ab9544e1bbe808397520b07ee
W20260807 18:31:22.745775 11420 leader_election.cc:341] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675): Not found: Tablet not found: b38ff64ab9544e1bbe808397520b07ee
I20260807 18:31:22.745944 11420 leader_election.cc:302] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4299a843b4049ceaa60048e64bebc80; no voters: 3bc7d47aab6f46a8bb48efc2d813dade, 841c868a39524c218efc67bbc5e46a53
I20260807 18:31:22.746196 11831 raft_consensus.cc:2726] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:31:22.902671 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:22.901888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:48553)
0807 18:31:22.902373 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:22.902485 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":255,"threads_started":1}
I20260807 18:31:23.356649 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:23.355922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:42005)
0807 18:31:23.356381 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:23.356488 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
I20260807 18:31:23.445667 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:23.444991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:47135)
0807 18:31:23.445401 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:23.445498 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":217,"threads_started":1}
I20260807 18:31:23.904249 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:23.903575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:52931)
0807 18:31:23.903998 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:23.904074 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":211,"threads_started":1}
I20260807 18:31:24.303514 11837 raft_consensus.cc:482] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:31:24.303759 11837 raft_consensus.cc:504] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:24.304494 11837 leader_election.cc:288] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141), 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
W20260807 18:31:24.305449 11417 leader_election.cc:341] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141): Not found: Tablet not found: b38ff64ab9544e1bbe808397520b07ee
W20260807 18:31:24.305604 11420 leader_election.cc:341] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675): Not found: Tablet not found: b38ff64ab9544e1bbe808397520b07ee
I20260807 18:31:24.305769 11420 leader_election.cc:302] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4299a843b4049ceaa60048e64bebc80; no voters: 3bc7d47aab6f46a8bb48efc2d813dade, 841c868a39524c218efc67bbc5e46a53
I20260807 18:31:24.306025 11837 raft_consensus.cc:2726] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:31:24.358446 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:24.357499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:49969)
0807 18:31:24.358114 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:24.358201 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":247,"threads_started":1}
I20260807 18:31:24.447631 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:24.446683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:40263)
0807 18:31:24.447291 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:24.447380 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":256,"threads_started":1}
I20260807 18:31:24.905999 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:24.905160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:35007)
0807 18:31:24.905713 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:24.905804 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":315,"threads_started":1}
I20260807 18:31:25.360064 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:25.359386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:54651)
0807 18:31:25.359812 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:25.359900 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260807 18:31:25.449267 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:25.448556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:56117)
0807 18:31:25.448988 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:25.449087 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":219,"threads_started":1}
I20260807 18:31:25.907648 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:25.906843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:48943)
0807 18:31:25.907378 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:25.907454 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":346,"threads_started":1}
I20260807 18:31:26.154134 11844 raft_consensus.cc:482] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:31:26.154364 11844 raft_consensus.cc:504] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:26.155463 11844 leader_election.cc:288] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141), 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
W20260807 18:31:26.156030 11417 leader_election.cc:341] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141): Not found: Tablet not found: b38ff64ab9544e1bbe808397520b07ee
W20260807 18:31:26.156214 11420 leader_election.cc:341] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675): Not found: Tablet not found: b38ff64ab9544e1bbe808397520b07ee
I20260807 18:31:26.156373 11420 leader_election.cc:302] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4299a843b4049ceaa60048e64bebc80; no voters: 3bc7d47aab6f46a8bb48efc2d813dade, 841c868a39524c218efc67bbc5e46a53
I20260807 18:31:26.156685 11844 raft_consensus.cc:2726] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:31:26.361660 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:26.360894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:52013)
0807 18:31:26.361397 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:26.361480 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":277,"threads_started":1}
I20260807 18:31:26.450896 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:26.450155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:43277)
0807 18:31:26.450577 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:26.450685 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260807 18:31:26.909423 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:26.908652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:34455)
0807 18:31:26.909169 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:26.909247 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":228,"threads_started":1}
I20260807 18:31:27.363756 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:27.362649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:56999)
0807 18:31:27.363439 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:27.363523 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":244,"threads_started":1}
I20260807 18:31:27.452533 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:27.451831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:46255)
0807 18:31:27.452271 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:27.452372 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260807 18:31:27.911034 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:27.910347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:35149)
0807 18:31:27.910758 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:27.910843 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":211,"threads_started":1}
I20260807 18:31:28.365487 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:28.364794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:34439)
0807 18:31:28.365214 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:28.365297 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":214,"threads_started":1}
I20260807 18:31:28.454295 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:28.453447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:35535)
0807 18:31:28.453953 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:28.454104 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":228,"threads_started":1}
I20260807 18:31:28.912696 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:28.911951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:52359)
0807 18:31:28.912452 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:28.912541 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":259,"threads_started":1}
I20260807 18:31:29.367379 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:29.366582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:45797)
0807 18:31:29.367068 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:29.367209 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":239,"threads_started":1}
I20260807 18:31:29.455981 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:29.455263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:34667)
0807 18:31:29.455735 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:29.455817 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":263,"threads_started":1}
I20260807 18:31:29.914340 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:29.913634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:50553)
0807 18:31:29.914074 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:29.914152 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
I20260807 18:31:30.369289 11859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:30.368492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:53411)
0807 18:31:30.369005 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:30.369099 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":259,"threads_started":1}
I20260807 18:31:30.457847 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:30.457066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:59859)
0807 18:31:30.457564 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:30.457655 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":262,"threads_started":1}
I20260807 18:31:30.916294 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:30.915532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:49783)
0807 18:31:30.916042 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:30.916133 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":252,"threads_started":1}
I20260807 18:31:31.371031 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:31.370249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:56859)
0807 18:31:31.370769 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:31.370860 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":256,"threads_started":1}
I20260807 18:31:31.459669 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:31.458873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:56431)
0807 18:31:31.459373 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:31.459464 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":274,"threads_started":1}
I20260807 18:31:31.918093 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:31.917367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:52697)
0807 18:31:31.917825 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:31.917902 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":250,"threads_started":1}
I20260807 18:31:32.372716 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:32.371971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:32881)
0807 18:31:32.372431 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:32.372532 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":245,"threads_started":1}
I20260807 18:31:32.461349 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:32.460587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:55641)
0807 18:31:32.461086 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:32.461173 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":263,"threads_started":1}
I20260807 18:31:32.805440 11867 raft_consensus.cc:482] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:31:32.805688 11867 raft_consensus.cc:504] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841c868a39524c218efc67bbc5e46a53" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34141 } } peers { permanent_uuid: "d4299a843b4049ceaa60048e64bebc80" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44151 } } peers { permanent_uuid: "3bc7d47aab6f46a8bb48efc2d813dade" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 42675 } }
I20260807 18:31:32.806423 11867 leader_election.cc:288] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141), 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675)
W20260807 18:31:32.807341 11417 leader_election.cc:341] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 841c868a39524c218efc67bbc5e46a53 (127.27.1.194:34141): Not found: Tablet not found: b38ff64ab9544e1bbe808397520b07ee
W20260807 18:31:32.807538 11420 leader_election.cc:341] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bc7d47aab6f46a8bb48efc2d813dade (127.27.1.195:42675): Not found: Tablet not found: b38ff64ab9544e1bbe808397520b07ee
I20260807 18:31:32.807724 11420 leader_election.cc:302] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4299a843b4049ceaa60048e64bebc80; no voters: 3bc7d47aab6f46a8bb48efc2d813dade, 841c868a39524c218efc67bbc5e46a53
I20260807 18:31:32.807991 11867 raft_consensus.cc:2726] T b38ff64ab9544e1bbe808397520b07ee P d4299a843b4049ceaa60048e64bebc80 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:31:32.920025 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:32.919190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:48887)
0807 18:31:32.919774 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:32.919862 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":341,"threads_started":1}
I20260807 18:31:33.374430 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:33.373656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:55593)
0807 18:31:33.374163 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:33.374254 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":268,"threads_started":1}
I20260807 18:31:33.462929 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:33.462245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:56241)
0807 18:31:33.462670 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:33.462751 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":225,"threads_started":1}
I20260807 18:31:33.921681 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:33.920880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:43525)
0807 18:31:33.921397 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:33.921479 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
I20260807 18:31:34.376205 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:34.375417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:46833)
0807 18:31:34.375911 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:34.376001 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":238,"threads_started":1}
I20260807 18:31:34.464905 11873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:34.464160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.193:38939)
0807 18:31:34.464618 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:34.464721 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":247,"threads_started":1}
I20260807 18:31:34.923661 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:34.922881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.194:56617)
0807 18:31:34.923386 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:34.923501 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":297,"threads_started":1}
I20260807 18:31:35.361896 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 11362
I20260807 18:31:35.377276 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:35.363075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.27.1.193:44151)
0807 18:31:35.363685 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:35.363693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:35.363720 (+ 27us) server_negotiation.cc:407] Connection header received
0807 18:31:35.363799 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:35.363805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:35.364018 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:35.364198 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:35.365608 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:35.366562 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:35.367987 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:35.368552 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:35.368703 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:35.371490 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:35.371501 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:35.371516 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:35.371548 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:35.374276 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:35.374969 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:35.374976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:35.374978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:35.375166 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:35.375553 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:35.375558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:35.375561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:35.375927 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:35.376236 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:35.376615 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 18:31:35.376905 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":269,"threads_started":1}
I20260807 18:31:35.378474 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:35.377483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:33219 (local address 127.27.1.195:59121)
0807 18:31:35.378138 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:35.378285 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":365,"threads_started":1}
I20260807 18:31:35.378793 11535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:35.472700 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 11362
I20260807 18:31:35.495096 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 11544
I20260807 18:31:35.508872 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:35.495972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.27.1.194:34141)
0807 18:31:35.496486 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:35.496490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:35.496506 (+ 16us) server_negotiation.cc:407] Connection header received
0807 18:31:35.496558 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:35.496561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:35.496618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:35.496736 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:35.498152 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:35.498951 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:35.500264 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:35.500701 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:35.500846 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:35.503655 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:35.503660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:35.503663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:35.503696 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:35.506146 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:35.506797 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:35.506804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:35.506808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:35.506957 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:35.507386 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:35.507392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:35.507397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:35.507659 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:35.507935 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:35.508254 (+ 319us) server_negotiation.cc:299] Negotiation successful
0807 18:31:35.508504 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":255,"threads_started":1}
I20260807 18:31:35.509620 11673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:35.595093 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 11544
I20260807 18:31:35.616761 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 11681
I20260807 18:31:35.631983 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:35.617619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.27.1.195:42675)
0807 18:31:35.618195 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:35.618200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:35.618216 (+ 16us) server_negotiation.cc:407] Connection header received
0807 18:31:35.618274 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:35.618277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:35.618344 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:35.618482 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:35.620040 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:35.620868 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:35.622118 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:35.622557 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:35.622699 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:35.625347 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:35.625352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:35.625355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:35.625389 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:35.629056 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:35.629726 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:35.629922 (+ 196us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:35.629929 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:35.630101 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:35.630494 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:35.630497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:35.630500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:35.630762 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:35.631034 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:35.631379 (+ 345us) server_negotiation.cc:299] Negotiation successful
0807 18:31:35.631556 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":280,"threads_started":1}
I20260807 18:31:35.632799 11810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:35.719728 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 11681
2026-08-07T18:31:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24775 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:31:35 dist-test-slave-1xrh krb5kdc[11885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:31:35 dist-test-slave-1xrh krb5kdc[11885](info): set up 2 sockets
Aug 07 18:31:35 dist-test-slave-1xrh krb5kdc[11885](info): commencing operation
krb5kdc: starting...
W20260807 18:31:36.922312 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.005s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:31:37 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:31:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:31:37.210920 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38497
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:38497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:37.559284 11898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:37.559648 11898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:37.559813 11898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:37.567723 11898 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 18:31:37.567873 11898 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 18:31:37.567963 11898 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 18:31:37.568046 11898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:31:37.568128 11898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:37.568275 11898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:37.568356 11898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:31:37.568428 11898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:31:37.579113 11898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:38497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:37.580530 11898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:37.581954 11898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:37.594215 11905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:37.594273 11907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:37.594273 11903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:37.595211 11898 server_base.cc:1034] running on GCE node
I20260807 18:31:37.596089 11898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:37.597687 11898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:37.598888 11898 hybrid_clock.cc:648] HybridClock initialized: now 1786127497598799 us; error 45 us; skew 500 ppm
I20260807 18:31:37.599344 11898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:37 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:37.607703 11898 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:31:37.609416 11898 webserver.cc:460] Webserver started at http://127.27.1.254:33951/ using document root <none> and password file <none>
I20260807 18:31:37.610167 11898 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:37.610267 11898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:37.610562 11898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:37.614027 11898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "ef766a57049e42e9a58120e94356a8fb"
format_stamp: "Formatted at 2026-08-07 18:31:37 on dist-test-slave-1xrh"
server_key: "7b1f66883245590bcd26511e47ab65ae"
server_key_iv: "d8946b390ed258936f2cd19fa316e606"
server_key_version: "encryptionkey@0"
I20260807 18:31:37.614843 11898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "ef766a57049e42e9a58120e94356a8fb"
format_stamp: "Formatted at 2026-08-07 18:31:37 on dist-test-slave-1xrh"
server_key: "7b1f66883245590bcd26511e47ab65ae"
server_key_iv: "d8946b390ed258936f2cd19fa316e606"
server_key_version: "encryptionkey@0"
I20260807 18:31:37.620731 11898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 18:31:37.624908 11914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:37.626542 11898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260807 18:31:37.626788 11898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "ef766a57049e42e9a58120e94356a8fb"
format_stamp: "Formatted at 2026-08-07 18:31:37 on dist-test-slave-1xrh"
server_key: "7b1f66883245590bcd26511e47ab65ae"
server_key_iv: "d8946b390ed258936f2cd19fa316e606"
server_key_version: "encryptionkey@0"
I20260807 18:31:37.627084 11898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:37.641525 11898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:37.645887 11898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:37.646191 11898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:37.667102 11898 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:38497
I20260807 18:31:37.667109 11966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:38497 every 8 connection(s)
I20260807 18:31:37.668660 11898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:31:37.672459 11967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:37.677860 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 11898
I20260807 18:31:37.678118 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:31:37.678933 27655 external_mini_cluster.cc:1428] Setting key 51354ca2186f7321e70c7b346d814f84
I20260807 18:31:37.683516 11967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Bootstrap starting.
I20260807 18:31:37.687433 11967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:37.688481 11967 log.cc:824] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:37.691636 11967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: No bootstrap required, opened a new log
Aug 07 18:31:37 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:37.699561 11967 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:31:37.699947 11967 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:37.700071 11967 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef766a57049e42e9a58120e94356a8fb, State: Initialized, Role: FOLLOWER
I20260807 18:31:37.700771 11967 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:31:37.701053 11967 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:31:37.701215 11967 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:31:37.701436 11967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:37.704820 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:37.681124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.27.1.254:38497)
0807 18:31:37.682057 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:37.682070 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:37.682115 (+ 45us) server_negotiation.cc:407] Connection header received
0807 18:31:37.684118 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:37.684144 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:37.684553 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:37.684929 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:37.686943 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:37.688566 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:37.690318 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:37.691078 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:37.691362 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:37.696165 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:37.696174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:37.696193 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:37.696236 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:37.699584 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:37.700534 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:37.700543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:37.700553 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:37.700793 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:37.701346 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:37.701351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:37.701355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:37.702190 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:37.702507 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:37.702929 (+ 422us) server_negotiation.cc:299] Negotiation successful
0807 18:31:37.703438 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":452,"threads_started":1}
I20260807 18:31:37.704953 11967 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:31:37.705554 11967 leader_election.cc:302] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef766a57049e42e9a58120e94356a8fb; no voters:
I20260807 18:31:37.706223 11967 leader_election.cc:288] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:31:37.706336 11972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:31:37.708455 11967 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:31:37.708552 11972 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 1 LEADER]: Becoming Leader. State: Replica: ef766a57049e42e9a58120e94356a8fb, State: Running, Role: LEADER
I20260807 18:31:37.709065 11972 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:31:37.715322 11975 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef766a57049e42e9a58120e94356a8fb. Latest consensus state: current_term: 1 leader_uuid: "ef766a57049e42e9a58120e94356a8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } } }
I20260807 18:31:37.715313 11974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef766a57049e42e9a58120e94356a8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } } }
I20260807 18:31:37.715780 11975 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: This master's current role is: LEADER
I20260807 18:31:37.715797 11974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: This master's current role is: LEADER
W20260807 18:31:37.721306 11987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 18:31:37.721446 11987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 18:31:37.721598 11989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:31:37.723362 11989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:31:37.731549 11989 catalog_manager.cc:1347] Generated new cluster ID: 8de8184198ed4def902a4b9e3f65724e
I20260807 18:31:37.731732 11989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:31:37.750133 11989 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:31:37.751711 11989 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:31:37.757429 11989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Generated new TSK 0
I20260807 18:31:37.758205 11989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:31:37.834863 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38497
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:38.169811 11995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:38.170092 11995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:38.170249 11995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:38.177317 11995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:38.177471 11995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:38.177659 11995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:31:38.187197 11995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:38.188493 11995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:38.189980 11995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:38.203220 12000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:38.203289 12003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:38.203737 12001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:38.204233 11995 server_base.cc:1034] running on GCE node
I20260807 18:31:38.204819 11995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:38.205734 11995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:38.206983 11995 hybrid_clock.cc:648] HybridClock initialized: now 1786127498206918 us; error 58 us; skew 500 ppm
I20260807 18:31:38.207345 11995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:38 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:38.211010 11995 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:31:38.212589 11995 webserver.cc:460] Webserver started at http://127.27.1.193:40463/ using document root <none> and password file <none>
I20260807 18:31:38.213241 11995 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:38.213349 11995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:38.213609 11995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:38.216691 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "aebad1a5083d4be89c6824b27908118a"
format_stamp: "Formatted at 2026-08-07 18:31:38 on dist-test-slave-1xrh"
server_key: "aa67e8f436a8e79f369c3fb899ce26d5"
server_key_iv: "32a50c8b5c67d6beb95982d437dfcd3c"
server_key_version: "encryptionkey@0"
I20260807 18:31:38.217491 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "aebad1a5083d4be89c6824b27908118a"
format_stamp: "Formatted at 2026-08-07 18:31:38 on dist-test-slave-1xrh"
server_key: "aa67e8f436a8e79f369c3fb899ce26d5"
server_key_iv: "32a50c8b5c67d6beb95982d437dfcd3c"
server_key_version: "encryptionkey@0"
I20260807 18:31:38.222518 11995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:31:38.225965 12012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:38.227389 11995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:31:38.227530 11995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "aebad1a5083d4be89c6824b27908118a"
format_stamp: "Formatted at 2026-08-07 18:31:38 on dist-test-slave-1xrh"
server_key: "aa67e8f436a8e79f369c3fb899ce26d5"
server_key_iv: "32a50c8b5c67d6beb95982d437dfcd3c"
server_key_version: "encryptionkey@0"
I20260807 18:31:38.227689 11995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:38.242863 11995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:38.246663 11995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:38.246907 11995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:38.247867 11995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:38.249200 11995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:38.249298 11995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:38.249392 11995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:38.249440 11995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:38.283180 11995 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:34155
I20260807 18:31:38.283246 12130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:34155 every 8 connection(s)
I20260807 18:31:38.284549 11995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:31:38.292426 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 11995
I20260807 18:31:38.292649 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:31:38.293349 27655 external_mini_cluster.cc:1428] Setting key 804dc2de1c82cdb51cb61592b3e40cff
Aug 07 18:31:38 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:38.306310 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:38.287217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:39343 (local address 127.27.1.254:38497)
0807 18:31:38.287680 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:38.287685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:38.288493 (+ 808us) server_negotiation.cc:407] Connection header received
0807 18:31:38.290297 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:38.290303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:38.290393 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:38.290523 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:38.292066 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:38.292894 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:38.294640 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:38.295086 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:38.295247 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:38.299362 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:38.299372 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:38.299380 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:38.299427 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:38.302989 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:38.303768 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:38.303776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:38.303795 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:38.303950 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:38.304358 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:38.304365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:38.304370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:38.304764 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:38.304920 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:38.305574 (+ 654us) server_negotiation.cc:299] Negotiation successful
0807 18:31:38.305848 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":252,"threads_started":1}
I20260807 18:31:38.306888 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:38.287624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:39343)
0807 18:31:38.288330 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:38.288374 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:38.290030 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:38.290754 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:38.290762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:38.291322 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:38.291860 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:38.291877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:38.293077 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:38.293081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:38.294485 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:38.294499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:38.295353 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:38.295362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:38.295686 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:38.296402 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:38.296436 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:38.299086 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:38.303191 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:38.303197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:38.303209 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:38.303624 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:38.304060 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:38.304063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:38.304065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:38.304241 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:38.304885 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:38.304890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:38.305178 (+ 288us) client_negotiation.cc:769] Sending connection context
0807 18:31:38.305386 (+ 208us) client_negotiation.cc:240] Negotiation successful
0807 18:31:38.305762 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":273,"threads_started":1}
I20260807 18:31:38.308605 12131 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38497
I20260807 18:31:38.308915 12131 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:38.309834 12131 heartbeater.cc:511] Master 127.27.1.254:38497 requested a full tablet report, sending...
I20260807 18:31:38.312106 11931 ts_manager.cc:194] Registered new tserver with Master: aebad1a5083d4be89c6824b27908118a (127.27.1.193:34155)
I20260807 18:31:38.313578 11931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:39343
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:31:38.362339 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38497
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:38.696457 12137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:38.696727 12137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:38.696820 12137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:38.703923 12137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:38.704025 12137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:38.704140 12137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:31:38.713629 12137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:38.714767 12137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:38.716035 12137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:38.729110 12146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:38.729125 12142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:38.729175 12143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:38.736675 12137 server_base.cc:1034] running on GCE node
I20260807 18:31:38.737332 12137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:38.738301 12137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:38.739532 12137 hybrid_clock.cc:648] HybridClock initialized: now 1786127498739483 us; error 43 us; skew 500 ppm
I20260807 18:31:38.739850 12137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:38 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:38.742945 12137 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:31:38.744558 12137 webserver.cc:460] Webserver started at http://127.27.1.194:38241/ using document root <none> and password file <none>
I20260807 18:31:38.745271 12137 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:38.745420 12137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:38.745740 12137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:38.748737 12137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "ddbf54c3d6184635acaeaf7ca5338792"
format_stamp: "Formatted at 2026-08-07 18:31:38 on dist-test-slave-1xrh"
server_key: "c910a5c1d8387683266f04bce2701379"
server_key_iv: "a83c4af1dd393da9825dd02911338a17"
server_key_version: "encryptionkey@0"
I20260807 18:31:38.749603 12137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "ddbf54c3d6184635acaeaf7ca5338792"
format_stamp: "Formatted at 2026-08-07 18:31:38 on dist-test-slave-1xrh"
server_key: "c910a5c1d8387683266f04bce2701379"
server_key_iv: "a83c4af1dd393da9825dd02911338a17"
server_key_version: "encryptionkey@0"
I20260807 18:31:38.754777 12137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 18:31:38.758203 12153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:38.759624 12137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260807 18:31:38.759796 12137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "ddbf54c3d6184635acaeaf7ca5338792"
format_stamp: "Formatted at 2026-08-07 18:31:38 on dist-test-slave-1xrh"
server_key: "c910a5c1d8387683266f04bce2701379"
server_key_iv: "a83c4af1dd393da9825dd02911338a17"
server_key_version: "encryptionkey@0"
I20260807 18:31:38.760010 12137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:38.815411 12137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:38.819296 12137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:38.819592 12137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:38.820626 12137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:38.821910 12137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:38.822013 12137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:38.822163 12137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:38.822239 12137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:38.855611 12137 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:38017
I20260807 18:31:38.855695 12275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:38017 every 8 connection(s)
I20260807 18:31:38.856942 12137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:31:38.860849 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 12137
I20260807 18:31:38.861054 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:31:38.861550 27655 external_mini_cluster.cc:1428] Setting key e33a8febf2125ca90c452e96c85a3953
Aug 07 18:31:38 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:38.878844 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:38.859572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:52195 (local address 127.27.1.254:38497)
0807 18:31:38.860060 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:38.860066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:38.860936 (+ 870us) server_negotiation.cc:407] Connection header received
0807 18:31:38.863045 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:38.863050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:38.863112 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:38.863252 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:38.865295 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:38.866010 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:38.868504 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:38.868932 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:38.869030 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:38.873112 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:38.873117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:38.873120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:38.873150 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:38.875846 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:38.876559 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:38.876567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:38.876570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:38.876703 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:38.877130 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:38.877138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:38.877141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:38.877414 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:38.877576 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:38.878230 (+ 654us) server_negotiation.cc:299] Negotiation successful
0807 18:31:38.878454 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260807 18:31:38.879832 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:38.860082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:52195)
0807 18:31:38.860774 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:38.860818 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:38.862762 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:38.863503 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:38.863516 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:38.864304 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:38.865110 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:38.865137 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:38.866171 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:38.866179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:38.868360 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:38.868378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:38.869159 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:38.869169 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:38.869481 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:38.870248 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:38.870282 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:38.872883 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:38.875997 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:38.876006 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:38.876021 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:38.876440 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:38.876817 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:38.876823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:38.876827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:38.877017 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:38.877545 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:38.877552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:38.877866 (+ 314us) client_negotiation.cc:769] Sending connection context
0807 18:31:38.878148 (+ 282us) client_negotiation.cc:240] Negotiation successful
0807 18:31:38.878684 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
I20260807 18:31:38.881326 12276 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38497
I20260807 18:31:38.881677 12276 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:38.882517 12276 heartbeater.cc:511] Master 127.27.1.254:38497 requested a full tablet report, sending...
I20260807 18:31:38.884188 11931 ts_manager.cc:194] Registered new tserver with Master: ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017)
I20260807 18:31:38.885447 11931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:52195
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:31:38.926568 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38497
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:31:39.260365 12283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:31:39.260622 12283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:31:39.260766 12283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:31:39.268028 12283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:31:39.268126 12283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:31:39.268229 12283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:31:39.277536 12283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:31:39.278677 12283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:31:39.279928 12283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:31:39.293319 12289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:39.293375 12288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:31:39.293401 12291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:31:39.294170 12283 server_base.cc:1034] running on GCE node
I20260807 18:31:39.294771 12283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:31:39.295699 12283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:31:39.296906 12283 hybrid_clock.cc:648] HybridClock initialized: now 1786127499296859 us; error 44 us; skew 500 ppm
I20260807 18:31:39.297206 12283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:31:39 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127499, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:31:39.300302 12283 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:31:39.301754 12283 webserver.cc:460] Webserver started at http://127.27.1.195:45303/ using document root <none> and password file <none>
I20260807 18:31:39.302404 12283 fs_manager.cc:359] Metadata directory not provided
I20260807 18:31:39.302546 12283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:31:39.302860 12283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:31:39.305743 12283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "0835388faabb4e5fb79613d6be2a85b6"
format_stamp: "Formatted at 2026-08-07 18:31:39 on dist-test-slave-1xrh"
server_key: "32a2c86021a2924ecd74bb2664b2dbfe"
server_key_iv: "c1f0986725af5c667b84b80601a7332d"
server_key_version: "encryptionkey@0"
I20260807 18:31:39.306499 12283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "0835388faabb4e5fb79613d6be2a85b6"
format_stamp: "Formatted at 2026-08-07 18:31:39 on dist-test-slave-1xrh"
server_key: "32a2c86021a2924ecd74bb2664b2dbfe"
server_key_iv: "c1f0986725af5c667b84b80601a7332d"
server_key_version: "encryptionkey@0"
I20260807 18:31:39.312773 12283 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260807 18:31:39.316498 12131 heartbeater.cc:503] Master 127.27.1.254:38497 was elected leader, sending a full tablet report...
I20260807 18:31:39.316625 12299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:39.318537 12283 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 18:31:39.318704 12283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "0835388faabb4e5fb79613d6be2a85b6"
format_stamp: "Formatted at 2026-08-07 18:31:39 on dist-test-slave-1xrh"
server_key: "32a2c86021a2924ecd74bb2664b2dbfe"
server_key_iv: "c1f0986725af5c667b84b80601a7332d"
server_key_version: "encryptionkey@0"
I20260807 18:31:39.318954 12283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:31:39.347926 12283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:31:39.351661 12283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:31:39.351920 12283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:31:39.352896 12283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:31:39.354157 12283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:31:39.354228 12283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:39.354314 12283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:31:39.354365 12283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:31:39.389079 12283 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:45379
I20260807 18:31:39.389133 12416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:45379 every 8 connection(s)
I20260807 18:31:39.390388 12283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:31:39.393143 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 12283
I20260807 18:31:39.393338 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:31:39.393944 27655 external_mini_cluster.cc:1428] Setting key 1888e24a0b88b864e75e910c4e98f1d4
Aug 07 18:31:39 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127499, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:31:39.410660 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.393127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:52795 (local address 127.27.1.254:38497)
0807 18:31:39.393688 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:39.393692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:39.394766 (+ 1074us) server_negotiation.cc:407] Connection header received
0807 18:31:39.396864 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:39.396870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:39.396994 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:39.397112 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:39.398621 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.399469 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.400956 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.401357 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.401456 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.404853 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:39.404858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:39.404861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:39.404897 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:39.407429 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:39.408132 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:39.408140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:39.408143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:39.408304 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:39.408743 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:39.408748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:39.408751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:39.409119 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:39.409314 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:39.410027 (+ 713us) server_negotiation.cc:299] Negotiation successful
0807 18:31:39.410267 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":333,"threads_started":1}
I20260807 18:31:39.411711 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.393744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:52795)
0807 18:31:39.394570 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:39.394629 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:39.396613 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:39.397405 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:39.397414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:39.397905 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:31:39.398474 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.398491 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.399598 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.399602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.400826 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.400838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.401604 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.401612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.401871 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.402476 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:31:39.402497 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:31:39.404675 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:31:39.407570 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:39.407580 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:39.407594 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:39.408008 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:39.408424 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:31:39.408430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:31:39.408433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:31:39.408618 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:31:39.409284 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:31:39.409292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:31:39.409660 (+ 368us) client_negotiation.cc:769] Sending connection context
0807 18:31:39.409924 (+ 264us) client_negotiation.cc:240] Negotiation successful
0807 18:31:39.410506 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":341,"threads_started":1}
I20260807 18:31:39.412986 12417 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38497
I20260807 18:31:39.413280 12417 heartbeater.cc:464] Registering TS with master...
I20260807 18:31:39.413975 12417 heartbeater.cc:511] Master 127.27.1.254:38497 requested a full tablet report, sending...
I20260807 18:31:39.415414 11931 ts_manager.cc:194] Registered new tserver with Master: 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379)
I20260807 18:31:39.416213 11931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:52795
I20260807 18:31:39.421452 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:31:39.438962 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.426025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.27.1.254:38497)
0807 18:31:39.426174 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:39.426178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:39.426646 (+ 468us) server_negotiation.cc:407] Connection header received
0807 18:31:39.426778 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:39.426782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:39.426850 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:39.426945 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:39.428374 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.429057 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.430383 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.430788 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.430932 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.433532 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:39.433537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:39.433540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:39.433570 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:39.436294 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:39.436993 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:39.437002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:39.437005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:39.437131 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:39.437509 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:39.437517 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:39.437520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:39.437826 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:39.438075 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:39.438409 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 18:31:39.438627 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 18:31:39.442845 11931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:31:39.449626 11931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:31:39.487668 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.461546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.27.1.195:45379)
0807 18:31:39.478421 (+ 16875us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:39.478429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:39.479048 (+ 619us) server_negotiation.cc:407] Connection header received
0807 18:31:39.479199 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:39.479210 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:39.479506 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:39.480618 (+ 1112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:39.480958 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.482814 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.485254 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.486196 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.486351 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.486596 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:39.487163 (+ 567us) server_negotiation.cc:299] Negotiation successful
0807 18:31:39.487326 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16692,"thread_start_us":16882,"threads_started":1}
I20260807 18:31:39.487859 12431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.460916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:38017 (local address 127.0.0.1:52066)
0807 18:31:39.476852 (+ 15936us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:39.476908 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:39.477174 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:39.478730 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:39.478738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:39.478771 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:39.479222 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.479239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.481847 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.481856 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.484744 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.484761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.486915 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.486936 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.487171 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.487214 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 18:31:39.487302 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 18:31:39.487446 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15593,"thread_start_us":16885,"threads_started":1}
I20260807 18:31:39.488399 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.461117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:45379 (local address 127.0.0.1:57648)
0807 18:31:39.478695 (+ 17578us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:39.478716 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:39.478873 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:39.479759 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:39.479767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:39.479802 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:39.480339 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.480353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.482956 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.482962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.485101 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.485124 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.486414 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.486420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.486644 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.486734 (+ 90us) client_negotiation.cc:769] Sending connection context
0807 18:31:39.486899 (+ 165us) client_negotiation.cc:240] Negotiation successful
0807 18:31:39.488046 (+ 1147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17453,"mutex_wait_us":887,"thread_start_us":16313,"threads_started":1}
I20260807 18:31:39.488647 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.461288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.27.1.194:38017)
0807 18:31:39.477831 (+ 16543us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:39.477841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:39.477869 (+ 28us) server_negotiation.cc:407] Connection header received
0807 18:31:39.478009 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:39.478019 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:39.478349 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:39.478551 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:39.479526 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.481630 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.484930 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.486423 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.486556 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.486752 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:39.488106 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0807 18:31:39.488290 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16376,"thread_start_us":16258,"threads_started":1}
I20260807 18:31:39.491235 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.479176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.0.0.1:35640)
0807 18:31:39.483013 (+ 3837us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:39.483030 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:39.483212 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:39.484084 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:39.484088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:39.484111 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:39.484576 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.484590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.486584 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.486589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.488923 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.488940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.490433 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.490439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.490608 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.490642 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 18:31:39.490732 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 18:31:39.490855 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3705,"thread_start_us":3519,"threads_started":1}
I20260807 18:31:39.492172 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.478599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.27.1.193:34155)
0807 18:31:39.482505 (+ 3906us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:39.482514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:39.483149 (+ 635us) server_negotiation.cc:407] Connection header received
0807 18:31:39.483412 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:39.483419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:39.483634 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:39.483832 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:39.484747 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.486394 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.489114 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.490220 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.490360 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.491647 (+ 1287us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:39.491723 (+ 76us) server_negotiation.cc:299] Negotiation successful
0807 18:31:39.491873 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3733,"thread_start_us":3585,"threads_started":1}
I20260807 18:31:39.492522 12205 tablet_service.cc:1467] Processing CreateTablet for tablet 34c509c5818046389e94091ee15a96b8 (DEFAULT_TABLE table=test-table [id=e7646a72f08049e2bd2460e6b5e43ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:39.492610 12351 tablet_service.cc:1467] Processing CreateTablet for tablet 34c509c5818046389e94091ee15a96b8 (DEFAULT_TABLE table=test-table [id=e7646a72f08049e2bd2460e6b5e43ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:39.493898 12351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c509c5818046389e94091ee15a96b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:39.493947 12205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c509c5818046389e94091ee15a96b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:39.495488 12062 tablet_service.cc:1467] Processing CreateTablet for tablet 34c509c5818046389e94091ee15a96b8 (DEFAULT_TABLE table=test-table [id=e7646a72f08049e2bd2460e6b5e43ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:31:39.496809 12062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c509c5818046389e94091ee15a96b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:31:39.505332 12438 tablet_bootstrap.cc:492] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: Bootstrap starting.
I20260807 18:31:39.508235 12439 tablet_bootstrap.cc:492] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: Bootstrap starting.
I20260807 18:31:39.508816 12438 tablet_bootstrap.cc:654] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:39.510797 12440 tablet_bootstrap.cc:492] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: Bootstrap starting.
I20260807 18:31:39.511652 12438 log.cc:824] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:39.512042 12439 tablet_bootstrap.cc:654] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:39.513087 12439 log.cc:824] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:39.514250 12440 tablet_bootstrap.cc:654] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: Neither blocks nor log segments found. Creating new log.
I20260807 18:31:39.514907 12438 tablet_bootstrap.cc:492] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: No bootstrap required, opened a new log
I20260807 18:31:39.515205 12438 ts_tablet_manager.cc:1397] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 18:31:39.515341 12440 log.cc:824] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: Log is configured to *not* fsync() on all Append() calls
I20260807 18:31:39.516124 12439 tablet_bootstrap.cc:492] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: No bootstrap required, opened a new log
I20260807 18:31:39.516374 12439 ts_tablet_manager.cc:1397] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260807 18:31:39.517727 12440 tablet_bootstrap.cc:492] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: No bootstrap required, opened a new log
I20260807 18:31:39.517988 12440 ts_tablet_manager.cc:1397] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260807 18:31:39.526407 12440 raft_consensus.cc:348] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:31:39.526798 12440 raft_consensus.cc:374] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:39.526901 12440 raft_consensus.cc:729] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebad1a5083d4be89c6824b27908118a, State: Initialized, Role: FOLLOWER
I20260807 18:31:39.526839 12438 raft_consensus.cc:348] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:31:39.526839 12439 raft_consensus.cc:348] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:31:39.527294 12438 raft_consensus.cc:374] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:39.527433 12439 raft_consensus.cc:374] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:31:39.527405 12438 raft_consensus.cc:729] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0835388faabb4e5fb79613d6be2a85b6, State: Initialized, Role: FOLLOWER
I20260807 18:31:39.527530 12439 raft_consensus.cc:729] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbf54c3d6184635acaeaf7ca5338792, State: Initialized, Role: FOLLOWER
I20260807 18:31:39.527648 12440 consensus_queue.cc:260] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:31:39.528002 12438 consensus_queue.cc:260] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:31:39.528172 12439 consensus_queue.cc:260] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:31:39.529532 12440 ts_tablet_manager.cc:1428] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260807 18:31:39.536548 12439 ts_tablet_manager.cc:1428] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
W20260807 18:31:39.536979 12132 tablet.cc:2369] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:39.537042 12276 heartbeater.cc:503] Master 127.27.1.254:38497 was elected leader, sending a full tablet report...
I20260807 18:31:39.537806 12417 heartbeater.cc:503] Master 127.27.1.254:38497 was elected leader, sending a full tablet report...
I20260807 18:31:39.538287 12438 ts_tablet_manager.cc:1428] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: Time spent starting tablet: real 0.023s user 0.018s sys 0.000s
I20260807 18:31:39.591647 12446 raft_consensus.cc:482] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:31:39.591980 12446 raft_consensus.cc:504] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:31:39.593668 12446 leader_election.cc:288] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379), aebad1a5083d4be89c6824b27908118a (127.27.1.193:34155)
I20260807 18:31:39.601078 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.594003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:45379 (local address 127.27.1.194:34335)
0807 18:31:39.594610 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:39.594626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:39.594873 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:39.595258 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:39.595263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:39.595287 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:39.595766 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.595777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.597031 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.597034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.599162 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.599177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.600444 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.600448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.600566 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.600589 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 18:31:39.600664 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 18:31:39.600768 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":22,"thread_start_us":310,"threads_started":1}
I20260807 18:31:39.601135 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.594042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:34335 (local address 127.27.1.195:45379)
0807 18:31:39.594290 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:39.594298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:39.594740 (+ 442us) server_negotiation.cc:407] Connection header received
0807 18:31:39.595034 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:39.595039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:39.595116 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:39.595243 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:39.595917 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.596906 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.599311 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.600243 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.600407 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.600488 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:39.600727 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 18:31:39.600875 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:31:39.602027 12372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c509c5818046389e94091ee15a96b8" candidate_uuid: "ddbf54c3d6184635acaeaf7ca5338792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0835388faabb4e5fb79613d6be2a85b6" is_pre_election: true
I20260807 18:31:39.602250 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.595216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:42685 (local address 127.27.1.193:34155)
0807 18:31:39.595561 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:39.595568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:39.595956 (+ 388us) server_negotiation.cc:407] Connection header received
0807 18:31:39.596095 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:39.596100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:39.596175 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:39.596269 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:31:39.596743 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.597517 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.599905 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.601138 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.601302 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.601384 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:39.601876 (+ 492us) server_negotiation.cc:299] Negotiation successful
0807 18:31:39.601998 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260807 18:31:39.602360 12448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.594709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.27.1.194:42685)
0807 18:31:39.595872 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0807 18:31:39.595887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:31:39.595999 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:31:39.596288 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:31:39.596292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:31:39.596309 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:31:39.596583 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.596590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.597653 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.597657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.599739 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:31:39.599753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.601530 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.601536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:31:39.601721 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.601743 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 18:31:39.601850 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 18:31:39.601982 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":53,"thread_start_us":835,"threads_started":1}
I20260807 18:31:39.602617 12372 raft_consensus.cc:2454] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbf54c3d6184635acaeaf7ca5338792 in term 0.
I20260807 18:31:39.603376 12156 leader_election.cc:302] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0835388faabb4e5fb79613d6be2a85b6, ddbf54c3d6184635acaeaf7ca5338792; no voters:
I20260807 18:31:39.603467 12082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c509c5818046389e94091ee15a96b8" candidate_uuid: "ddbf54c3d6184635acaeaf7ca5338792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aebad1a5083d4be89c6824b27908118a" is_pre_election: true
I20260807 18:31:39.603757 12446 raft_consensus.cc:2781] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:31:39.603883 12446 raft_consensus.cc:482] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:31:39.603952 12446 raft_consensus.cc:3037] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:39.603971 12082 raft_consensus.cc:2454] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbf54c3d6184635acaeaf7ca5338792 in term 0.
I20260807 18:31:39.606263 12446 raft_consensus.cc:504] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:31:39.606784 12446 leader_election.cc:288] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [CANDIDATE]: Term 1 election: Requested vote from peers 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379), aebad1a5083d4be89c6824b27908118a (127.27.1.193:34155)
I20260807 18:31:39.607244 12372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c509c5818046389e94091ee15a96b8" candidate_uuid: "ddbf54c3d6184635acaeaf7ca5338792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0835388faabb4e5fb79613d6be2a85b6"
I20260807 18:31:39.607347 12082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c509c5818046389e94091ee15a96b8" candidate_uuid: "ddbf54c3d6184635acaeaf7ca5338792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aebad1a5083d4be89c6824b27908118a"
I20260807 18:31:39.607452 12372 raft_consensus.cc:3037] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:31:39.607579 12082 raft_consensus.cc:3037] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a [term 0 FOLLOWER]: Advancing to term 1
W20260807 18:31:39.608952 12277 tablet.cc:2369] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:39.610239 12082 raft_consensus.cc:2454] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbf54c3d6184635acaeaf7ca5338792 in term 1.
I20260807 18:31:39.610247 12372 raft_consensus.cc:2454] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbf54c3d6184635acaeaf7ca5338792 in term 1.
I20260807 18:31:39.610625 12156 leader_election.cc:302] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aebad1a5083d4be89c6824b27908118a, ddbf54c3d6184635acaeaf7ca5338792; no voters:
I20260807 18:31:39.610946 12446 raft_consensus.cc:2781] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:31:39.611510 12446 raft_consensus.cc:686] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 1 LEADER]: Becoming Leader. State: Replica: ddbf54c3d6184635acaeaf7ca5338792, State: Running, Role: LEADER
I20260807 18:31:39.612056 12446 consensus_queue.cc:237] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:31:39.616724 11930 catalog_manager.cc:5495] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194). New cstate: current_term: 1 leader_uuid: "ddbf54c3d6184635acaeaf7ca5338792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } health_report { overall_health: UNKNOWN } } }
W20260807 18:31:39.642382 12419 tablet.cc:2369] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:31:39.663825 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.658085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.27.1.194:38017)
0807 18:31:39.658347 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:39.658355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:39.658376 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:31:39.658662 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:39.658666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:39.658748 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:39.658925 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:31:39.659388 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.660124 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.661614 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.662025 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.662161 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.662661 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:31:39.663221 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:31:39.663383 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:39.663437 (+ 54us) server_negotiation.cc:299] Negotiation successful
0807 18:31:39.663520 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 18:31:39.672046 12082 raft_consensus.cc:1261] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Refusing update from remote peer ddbf54c3d6184635acaeaf7ca5338792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:31:39.672174 12372 raft_consensus.cc:1261] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6 [term 1 FOLLOWER]: Refusing update from remote peer ddbf54c3d6184635acaeaf7ca5338792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:31:39.673043 12446 consensus_queue.cc:1035] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:31:39.673441 12450 consensus_queue.cc:1035] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:31:39.688494 12454 mvcc.cc:205] Tried to move back new op lower bound from 7315978238646525952 to 7315978238415249408. Current Snapshot: MvccSnapshot[applied={T|T < 7315978238646525952}]
I20260807 18:31:39.690284 12455 mvcc.cc:205] Tried to move back new op lower bound from 7315978238646525952 to 7315978238415249408. Current Snapshot: MvccSnapshot[applied={T|T < 7315978238646525952}]
I20260807 18:31:39.705876 11930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:31:39.706157 11930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:31:39.709730 11930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Sending DeleteTablet for 3 replicas of tablet 34c509c5818046389e94091ee15a96b8
I20260807 18:31:39.711117 12205 tablet_service.cc:1514] Processing DeleteTablet for tablet 34c509c5818046389e94091ee15a96b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:39 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:52066
I20260807 18:31:39.711146 12351 tablet_service.cc:1514] Processing DeleteTablet for tablet 34c509c5818046389e94091ee15a96b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:39 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:57648
I20260807 18:31:39.711261 12062 tablet_service.cc:1514] Processing DeleteTablet for tablet 34c509c5818046389e94091ee15a96b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:31:39 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:35640
I20260807 18:31:39.712592 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 11995
I20260807 18:31:39.715708 12464 tablet_replica.cc:330] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: stopping tablet replica
I20260807 18:31:39.721978 12462 tablet_replica.cc:330] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: stopping tablet replica
I20260807 18:31:39.722596 12462 raft_consensus.cc:2229] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:31:39.723537 12464 raft_consensus.cc:2229] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:31:39.723616 12462 raft_consensus.cc:2258] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:39.724195 12464 raft_consensus.cc:2258] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:39.726105 12462 ts_tablet_manager.cc:1905] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:31:39.728302 12464 ts_tablet_manager.cc:1905] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:31:39.733687 12463 tablet_replica.cc:330] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: stopping tablet replica
I20260807 18:31:39.734072 12463 raft_consensus.cc:2229] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:31:39.734516 12463 raft_consensus.cc:2258] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:31:39.736076 12462 ts_tablet_manager.cc:1918] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:31:39.737293 12463 ts_tablet_manager.cc:1905] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 18:31:39 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:31:39.743433 12464 ts_tablet_manager.cc:1918] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:31:39.743754 12462 log.cc:1196] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/34c509c5818046389e94091ee15a96b8
I20260807 18:31:39.743993 12464 log.cc:1196] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/34c509c5818046389e94091ee15a96b8
I20260807 18:31:39.744688 12462 ts_tablet_manager.cc:1939] T 34c509c5818046389e94091ee15a96b8 P ddbf54c3d6184635acaeaf7ca5338792: Deleting consensus metadata
I20260807 18:31:39.744762 12463 ts_tablet_manager.cc:1918] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:31:39.744994 12463 log.cc:1196] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/34c509c5818046389e94091ee15a96b8
I20260807 18:31:39.745304 12464 ts_tablet_manager.cc:1939] T 34c509c5818046389e94091ee15a96b8 P aebad1a5083d4be89c6824b27908118a: Deleting consensus metadata
I20260807 18:31:39.745601 12463 ts_tablet_manager.cc:1939] T 34c509c5818046389e94091ee15a96b8 P 0835388faabb4e5fb79613d6be2a85b6: Deleting consensus metadata
I20260807 18:31:39.747655 11917 catalog_manager.cc:4841] TS 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379): tablet 34c509c5818046389e94091ee15a96b8 (table test-table [id=e7646a72f08049e2bd2460e6b5e43ab6]) successfully deleted
I20260807 18:31:39.748955 11916 catalog_manager.cc:4841] TS ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017): tablet 34c509c5818046389e94091ee15a96b8 (table test-table [id=e7646a72f08049e2bd2460e6b5e43ab6]) successfully deleted
I20260807 18:31:39.749104 11917 catalog_manager.cc:4841] TS aebad1a5083d4be89c6824b27908118a (127.27.1.193:34155): tablet 34c509c5818046389e94091ee15a96b8 (table test-table [id=e7646a72f08049e2bd2460e6b5e43ab6]) successfully deleted
I20260807 18:31:39.749778 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.714060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.27.1.193:34155)
0807 18:31:39.714290 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:39.714299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:39.725289 (+ 10990us) server_negotiation.cc:407] Connection header received
0807 18:31:39.725385 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:39.725393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:39.725520 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:39.725657 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:39.728410 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.729887 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.734927 (+ 5040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.735452 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.735533 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.741976 (+ 6443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:39.741994 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:39.742012 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:39.742059 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:39.746308 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:39.747321 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:39.747332 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:39.747338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:39.747522 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:39.748019 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:39.748024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:39.748027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:39.748428 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:39.748632 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:39.749327 (+ 695us) server_negotiation.cc:299] Negotiation successful
0807 18:31:39.749517 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:31:39.751075 12126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:39.845460 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 11995
I20260807 18:31:39.868645 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 12137
Aug 07 18:31:39 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:31:39.885257 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:39.869920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.27.1.194:38017)
0807 18:31:39.870081 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:39.870086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:39.870102 (+ 16us) server_negotiation.cc:407] Connection header received
0807 18:31:39.870169 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:39.870173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:39.870231 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:39.870363 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:39.871740 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.872603 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.873988 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:39.874429 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:39.874570 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:39.879062 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:39.879073 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:39.879086 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:39.879166 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:39.882229 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:39.882949 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:39.882960 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:39.882962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:39.883112 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:39.883575 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:39.883580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:39.883583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:39.883966 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:39.884333 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:39.884606 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 18:31:39.884841 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 18:31:39.886041 12270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:40.002692 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 12137
I20260807 18:31:40.025589 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 12283
Aug 07 18:31:40 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:31:40.041283 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:31:40.026380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.27.1.195:45379)
0807 18:31:40.026598 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 18:31:40.026602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:31:40.026622 (+ 20us) server_negotiation.cc:407] Connection header received
0807 18:31:40.026707 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:31:40.026710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:31:40.026775 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:31:40.026862 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:31:40.028491 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:40.029231 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:40.030485 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:31:40.030860 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:31:40.030954 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:31:40.035678 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:31:40.035692 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:31:40.035710 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:31:40.035754 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:31:40.038345 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:40.039062 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:40.039069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:40.039071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:40.039305 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:31:40.039671 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:31:40.039676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:31:40.039679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:31:40.040047 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:31:40.040361 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 18:31:40.040699 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 18:31:40.040936 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260807 18:31:40.041958 12412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:40.132722 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 12283
I20260807 18:31:40.153255 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 11898
I20260807 18:31:40.154428 11962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:31:40.266114 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 11898
I20260807 18:32:00.292460 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38497
--webserver_interface=127.27.1.254
--webserver_port=33951
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:38497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:00.637014 12471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:00.637310 12471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:00.637468 12471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:00.644724 12471 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 18:32:00.644820 12471 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 18:32:00.644891 12471 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 18:32:00.644997 12471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:32:00.645069 12471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:00.645170 12471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:00.645222 12471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:32:00.645314 12471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:32:00.656586 12471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:38497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=33951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:00.657871 12471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:00.659291 12471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:00.685181 12477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:00.685266 12480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:00.685266 12478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:00.685894 12471 server_base.cc:1034] running on GCE node
I20260807 18:32:00.686760 12471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:00.688373 12471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:00.689636 12471 hybrid_clock.cc:648] HybridClock initialized: now 1786127520689569 us; error 47 us; skew 500 ppm
I20260807 18:32:00.690044 12471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:00 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127520, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:00.701563 12471 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:32:00.703460 12471 webserver.cc:460] Webserver started at http://127.27.1.254:33951/ using document root <none> and password file <none>
I20260807 18:32:00.704311 12471 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:00.704471 12471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:00.710515 12471 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260807 18:32:00.713474 12487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:00.714818 12471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 18:32:00.715022 12471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "ef766a57049e42e9a58120e94356a8fb"
format_stamp: "Formatted at 2026-08-07 18:31:37 on dist-test-slave-1xrh"
server_key: "7b1f66883245590bcd26511e47ab65ae"
server_key_iv: "d8946b390ed258936f2cd19fa316e606"
server_key_version: "encryptionkey@0"
I20260807 18:32:00.716135 12471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:00.739293 12471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:00.743932 12471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:00.744233 12471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:00.764859 12471 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:38497
I20260807 18:32:00.764902 12539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:38497 every 8 connection(s)
I20260807 18:32:00.766413 12471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:32:00.773708 12540 sys_catalog.cc:263] Verifying existing consensus state
I20260807 18:32:00.774269 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 12471
I20260807 18:32:00.775209 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:34155
--local_ip_for_outbound_sockets=127.27.1.193
--tserver_master_addrs=127.27.1.254:38497
--webserver_port=40463
--webserver_interface=127.27.1.193
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:32:00.777210 12540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Bootstrap starting.
I20260807 18:32:00.800897 12540 log.cc:824] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:00.817878 12540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 18:32:00.818579 12540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Bootstrap complete.
I20260807 18:32:00.829665 12540 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:32:00.830555 12540 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef766a57049e42e9a58120e94356a8fb, State: Initialized, Role: FOLLOWER
I20260807 18:32:00.831293 12540 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:32:00.831555 12540 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:32:00.831684 12540 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:32:00.831881 12540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 1 FOLLOWER]: Advancing to term 2
I20260807 18:32:00.835856 12540 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:32:00.836373 12540 leader_election.cc:302] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef766a57049e42e9a58120e94356a8fb; no voters:
I20260807 18:32:00.837074 12540 leader_election.cc:288] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 18:32:00.837203 12544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 2 FOLLOWER]: Leader election won for term 2
I20260807 18:32:00.838352 12544 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 2 LEADER]: Becoming Leader. State: Replica: ef766a57049e42e9a58120e94356a8fb, State: Running, Role: LEADER
I20260807 18:32:00.838966 12544 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:32:00.839455 12540 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:32:00.841737 12546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef766a57049e42e9a58120e94356a8fb. Latest consensus state: current_term: 2 leader_uuid: "ef766a57049e42e9a58120e94356a8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } } }
I20260807 18:32:00.842182 12546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:00.842660 12545 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef766a57049e42e9a58120e94356a8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } } }
I20260807 18:32:00.843026 12545 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:00.851930 12550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:32:00.861567 12550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:32:00.864264 12550 catalog_manager.cc:1259] Loaded cluster ID: 8de8184198ed4def902a4b9e3f65724e
I20260807 18:32:00.864459 12550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:32:00.869895 12550 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:32:00.871948 12550 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Loaded TSK: 0
I20260807 18:32:00.907341 12550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Generated new TSK 1
I20260807 18:32:00.907758 12550 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 18:32:01.180817 12542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:01.181099 12542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:01.181244 12542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:01.188645 12542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:01.188759 12542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:01.188970 12542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:32:01.199154 12542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:34155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=40463
--tserver_master_addrs=127.27.1.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:01.200527 12542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:01.201856 12542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:01.216935 12570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:01.217478 12569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:01.218472 12572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:01.218755 12542 server_base.cc:1034] running on GCE node
I20260807 18:32:01.219602 12542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:01.220656 12542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:01.221944 12542 hybrid_clock.cc:648] HybridClock initialized: now 1786127521221881 us; error 52 us; skew 500 ppm
I20260807 18:32:01.222261 12542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:01 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:01.225731 12542 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:32:01.227412 12542 webserver.cc:460] Webserver started at http://127.27.1.193:40463/ using document root <none> and password file <none>
I20260807 18:32:01.228111 12542 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:01.228271 12542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:01.233825 12542 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 18:32:01.236697 12580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:01.238106 12542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 18:32:01.238281 12542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "aebad1a5083d4be89c6824b27908118a"
format_stamp: "Formatted at 2026-08-07 18:31:38 on dist-test-slave-1xrh"
server_key: "aa67e8f436a8e79f369c3fb899ce26d5"
server_key_iv: "32a50c8b5c67d6beb95982d437dfcd3c"
server_key_version: "encryptionkey@0"
I20260807 18:32:01.239315 12542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:01.269552 12542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:01.273646 12542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:01.273948 12542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:01.274945 12542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:01.276315 12542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:01.276413 12542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:01.276543 12542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:01.276640 12542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:01.311584 12542 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:34155
I20260807 18:32:01.311693 12697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:34155 every 8 connection(s)
I20260807 18:32:01.313620 12542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:32:01.321743 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 12542
I20260807 18:32:01.322597 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:38017
--local_ip_for_outbound_sockets=127.27.1.194
--tserver_master_addrs=127.27.1.254:38497
--webserver_port=38241
--webserver_interface=127.27.1.194
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:32:01 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:01.341214 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:01.317191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:56563 (local address 127.27.1.254:38497)
0807 18:32:01.318225 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:01.318270 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:01.319269 (+ 999us) server_negotiation.cc:407] Connection header received
0807 18:32:01.321400 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:01.321421 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:01.321883 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:01.322268 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:01.324055 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:01.326086 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:01.327966 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:01.329035 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:01.329238 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:01.333273 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:01.333286 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:01.333308 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:01.333360 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:01.336646 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:01.337409 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:01.337430 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:01.337435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:01.337593 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:01.338038 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:01.338043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:01.338047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:01.338518 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:01.338730 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:01.339601 (+ 871us) server_negotiation.cc:299] Negotiation successful
0807 18:32:01.340024 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":326,"threads_started":1}
I20260807 18:32:01.341209 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:01.317373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:56563)
0807 18:32:01.318599 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:01.318638 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:01.320779 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:01.322427 (+ 1648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:01.322435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:01.323122 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:01.323807 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:01.323840 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:01.326272 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:01.326276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:01.327821 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:01.327834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:01.329252 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:01.329256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:01.329448 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:01.330195 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:01.330221 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:01.332961 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:01.336810 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:01.336819 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:01.336833 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:01.337260 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:01.337714 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:01.337719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:01.337723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:01.337907 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:01.338661 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:01.338669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:01.339016 (+ 347us) client_negotiation.cc:769] Sending connection context
0807 18:32:01.339344 (+ 328us) client_negotiation.cc:240] Negotiation successful
0807 18:32:01.339878 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":664,"threads_started":1}
I20260807 18:32:01.344906 12698 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38497
I20260807 18:32:01.345227 12698 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:01.345927 12698 heartbeater.cc:511] Master 127.27.1.254:38497 requested a full tablet report, sending...
I20260807 18:32:01.348490 12504 ts_manager.cc:194] Registered new tserver with Master: aebad1a5083d4be89c6824b27908118a (127.27.1.193:34155)
I20260807 18:32:01.351114 12504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:56563
W20260807 18:32:01.669126 12702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:01.669394 12702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:01.669490 12702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:01.676489 12702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:01.676596 12702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:01.676714 12702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:32:01.686177 12702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:38017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=38241
--tserver_master_addrs=127.27.1.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:01.687386 12702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:01.688637 12702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:01.701622 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:01.701647 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:01.701651 12708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:01.702886 12702 server_base.cc:1034] running on GCE node
I20260807 18:32:01.703521 12702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:01.704429 12702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:01.705658 12702 hybrid_clock.cc:648] HybridClock initialized: now 1786127521705626 us; error 41 us; skew 500 ppm
I20260807 18:32:01.705966 12702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:01 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:01.709204 12702 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:32:01.710551 12702 webserver.cc:460] Webserver started at http://127.27.1.194:38241/ using document root <none> and password file <none>
I20260807 18:32:01.711277 12702 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:01.711434 12702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:01.716636 12702 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:32:01.719295 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:01.720628 12702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:32:01.720804 12702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "ddbf54c3d6184635acaeaf7ca5338792"
format_stamp: "Formatted at 2026-08-07 18:31:38 on dist-test-slave-1xrh"
server_key: "c910a5c1d8387683266f04bce2701379"
server_key_iv: "a83c4af1dd393da9825dd02911338a17"
server_key_version: "encryptionkey@0"
I20260807 18:32:01.721771 12702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:01.732405 12702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:01.736733 12702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:01.737015 12702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:01.738016 12702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:01.739467 12702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:01.739602 12702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:01.739730 12702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:01.739827 12702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:01.774655 12702 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:38017
I20260807 18:32:01.774766 12836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:38017 every 8 connection(s)
I20260807 18:32:01.776122 12702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:32:01.779632 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 12702
I20260807 18:32:01.780412 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:45379
--local_ip_for_outbound_sockets=127.27.1.195
--tserver_master_addrs=127.27.1.254:38497
--webserver_port=45303
--webserver_interface=127.27.1.195
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 18:32:01 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:01.796159 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:01.778873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:60243 (local address 127.27.1.254:38497)
0807 18:32:01.779086 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:01.779092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:01.780359 (+ 1267us) server_negotiation.cc:407] Connection header received
0807 18:32:01.782459 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:01.782465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:01.782595 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:01.782709 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:01.784246 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:01.784963 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:01.786489 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:01.786947 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:01.787087 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:01.790468 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:01.790473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:01.790477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:01.790511 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:01.793107 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:01.793833 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:01.793841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:01.793849 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:01.793968 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:01.794397 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:01.794402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:01.794407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:01.794678 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:01.794838 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:01.795567 (+ 729us) server_negotiation.cc:299] Negotiation successful
0807 18:32:01.795793 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:32:01.797109 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:01.779306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:60243)
0807 18:32:01.780195 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:01.780246 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:01.782195 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:01.782944 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:01.782952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:01.783528 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:01.784077 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:01.784094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:01.785131 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:01.785143 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:01.786339 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:01.786350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:01.787216 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:01.787224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:01.787441 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:01.788119 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:01.788141 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:01.790243 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:01.793251 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:01.793260 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:01.793274 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:01.793703 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:01.794086 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:01.794091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:01.794095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:01.794277 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:01.794817 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:01.794824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:01.795189 (+ 365us) client_negotiation.cc:769] Sending connection context
0807 18:32:01.795459 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 18:32:01.795973 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":423,"threads_started":1}
I20260807 18:32:01.798588 12837 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38497
I20260807 18:32:01.798930 12837 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:01.799871 12837 heartbeater.cc:511] Master 127.27.1.254:38497 requested a full tablet report, sending...
I20260807 18:32:01.801614 12504 ts_manager.cc:194] Registered new tserver with Master: ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017)
I20260807 18:32:01.802874 12504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:60243
W20260807 18:32:02.121284 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:02.121554 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:02.121651 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:02.128895 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:02.128997 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:02.129109 12840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:32:02.138536 12840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:45379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=45303
--tserver_master_addrs=127.27.1.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:02.139863 12840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:02.141206 12840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:02.154590 12845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:02.155586 12840 server_base.cc:1034] running on GCE node
W20260807 18:32:02.158519 12849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:02.158865 12846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:02.159636 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:02.160663 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:02.161911 12840 hybrid_clock.cc:648] HybridClock initialized: now 1786127522161877 us; error 29 us; skew 500 ppm
I20260807 18:32:02.162233 12840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:02 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:02.165477 12840 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:32:02.167006 12840 webserver.cc:460] Webserver started at http://127.27.1.195:45303/ using document root <none> and password file <none>
I20260807 18:32:02.167754 12840 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:02.167950 12840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:02.173410 12840 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 18:32:02.176138 12859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:02.177464 12840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:32:02.177642 12840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "0835388faabb4e5fb79613d6be2a85b6"
format_stamp: "Formatted at 2026-08-07 18:31:39 on dist-test-slave-1xrh"
server_key: "32a2c86021a2924ecd74bb2664b2dbfe"
server_key_iv: "c1f0986725af5c667b84b80601a7332d"
server_key_version: "encryptionkey@0"
I20260807 18:32:02.178602 12840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:02.203953 12840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:02.207931 12840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:02.208221 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:02.209214 12840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:02.210628 12840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:02.210763 12840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:02.210892 12840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:02.210983 12840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:02.247820 12840 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:45379
I20260807 18:32:02.247879 12975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:45379 every 8 connection(s)
I20260807 18:32:02.249404 12840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:32:02.253257 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 12840
Aug 07 18:32:02 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:02.277738 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.254343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.27.1.254:38497)
0807 18:32:02.254879 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.254887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.256164 (+ 1277us) server_negotiation.cc:407] Connection header received
0807 18:32:02.256267 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.256274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.256371 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.256507 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:02.259106 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.260622 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.262614 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.263608 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.263777 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.267964 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:02.267972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:02.267979 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:02.268040 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:02.272250 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:02.274349 (+ 2099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:02.274355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:02.274359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:02.274524 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:02.275054 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:02.275059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:02.275062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:02.275522 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:02.275853 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.276935 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.277209 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":398,"thread_start_us":287,"threads_started":1}
I20260807 18:32:02.278162 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.252015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:42315 (local address 127.27.1.254:38497)
0807 18:32:02.252214 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.252219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.253746 (+ 1527us) server_negotiation.cc:407] Connection header received
0807 18:32:02.256846 (+ 3100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.256853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.256953 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.257058 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:02.259386 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.260212 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.262657 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.263190 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.263336 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.269293 (+ 5957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:02.269299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:02.269303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:02.269373 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:02.273758 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:02.274774 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:02.274780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:02.274784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:02.274945 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:02.275663 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:02.275668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:02.275670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:02.276090 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:02.276248 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.277261 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.277508 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":70}
I20260807 18:32:02.280575 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.252437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:42315)
0807 18:32:02.253549 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.253611 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:02.256504 (+ 2893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:02.257543 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:02.257558 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:02.258336 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:02.259207 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.259240 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.260463 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.260469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.262486 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.262502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.263431 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.263439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.263752 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.264815 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:02.264847 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:02.268712 (+ 3865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:02.273951 (+ 5239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:02.273966 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:02.273980 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:02.274591 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:02.275141 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:02.275146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:02.275170 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:02.275454 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:02.276256 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:02.276264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:02.276785 (+ 521us) client_negotiation.cc:769] Sending connection context
0807 18:32:02.277808 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0807 18:32:02.278430 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":453,"threads_started":1}
I20260807 18:32:02.284020 12976 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38497
I20260807 18:32:02.284370 12976 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:02.285320 12976 heartbeater.cc:511] Master 127.27.1.254:38497 requested a full tablet report, sending...
I20260807 18:32:02.287158 12504 ts_manager.cc:194] Registered new tserver with Master: 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379)
I20260807 18:32:02.288247 12504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:42315
I20260807 18:32:02.294036 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:32:02.302440 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.295306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.27.1.254:38497)
0807 18:32:02.295603 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.295611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.296102 (+ 491us) server_negotiation.cc:407] Connection header received
0807 18:32:02.296337 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.296344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.296488 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.296635 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:32:02.297095 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.297800 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.300135 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.300813 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.300947 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.301491 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:32:02.301754 (+ 263us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 18:32:02.302140 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55734: token expired
Metrics: {"server-negotiator.queue_time_us":90}
W20260807 18:32:02.302565 12979 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55734: token expired
W20260807 18:32:02.303332 12422 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.1.254:38497)
I20260807 18:32:02.318560 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.304032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.27.1.254:38497)
0807 18:32:02.304190 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.304195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.304211 (+ 16us) server_negotiation.cc:407] Connection header received
0807 18:32:02.304329 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.304335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.304403 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.304487 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:02.306001 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.307212 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.308783 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.309293 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.309415 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.312897 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:02.312902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:02.312906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:02.312938 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:02.315975 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:02.316733 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:02.316737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:02.316740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:02.316901 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:02.317296 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:02.317299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:02.317301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:02.317593 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:02.317785 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.318051 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.318204 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 18:32:02.322286 12504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:32:02.329814 12504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:32:02.354713 12698 heartbeater.cc:503] Master 127.27.1.254:38497 was elected leader, sending a full tablet report...
I20260807 18:32:02.366010 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.341850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:45379 (local address 127.0.0.1:55466)
0807 18:32:02.354987 (+ 13137us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.355050 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:02.355266 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:02.356674 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:02.356686 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:02.356718 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:02.357323 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.357340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.360072 (+ 2732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.360079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.362646 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.362660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.364691 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.364697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.365011 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.365176 (+ 165us) client_negotiation.cc:769] Sending connection context
0807 18:32:02.365328 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 18:32:02.365529 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12835,"mutex_wait_us":68,"thread_start_us":12626,"threads_started":1}
I20260807 18:32:02.366556 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.342349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.27.1.194:38017)
0807 18:32:02.355706 (+ 13357us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.355717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.356286 (+ 569us) server_negotiation.cc:407] Connection header received
0807 18:32:02.356404 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.356412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.356645 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.356866 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:02.358038 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.359740 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.363520 (+ 3780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.365094 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.365210 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.365417 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.365843 (+ 426us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.366024 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13169,"thread_start_us":11720,"threads_started":1}
I20260807 18:32:02.366724 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.341848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:38017 (local address 127.0.0.1:59818)
0807 18:32:02.355962 (+ 14114us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.355977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:02.356141 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:02.357150 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:02.357158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:02.357192 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:02.357783 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.357810 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.359913 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.359919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.363346 (+ 3427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.363362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.365445 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.365451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.365612 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.365658 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 18:32:02.365754 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 18:32:02.365885 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13995,"mutex_wait_us":114,"thread_start_us":13777,"threads_started":1}
I20260807 18:32:02.367067 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.342502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.27.1.195:45379)
0807 18:32:02.355962 (+ 13460us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.355972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.356011 (+ 39us) server_negotiation.cc:407] Connection header received
0807 18:32:02.356142 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.356149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.356407 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.356666 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:02.357643 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.359649 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.362836 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.364446 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.364597 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.364847 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.365396 (+ 549us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.365566 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13254,"thread_start_us":12665,"threads_started":1}
I20260807 18:32:02.367424 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.356480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.0.0.1:51960)
0807 18:32:02.357539 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.357556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:02.357689 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:02.358602 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:02.358610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:02.358634 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:02.358937 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.358944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.360524 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.360529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.363027 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.363043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.364774 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.364779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.365015 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.365176 (+ 161us) client_negotiation.cc:769] Sending connection context
0807 18:32:02.366039 (+ 863us) client_negotiation.cc:240] Negotiation successful
0807 18:32:02.366462 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":121,"thread_start_us":709,"threads_started":1}
I20260807 18:32:02.368499 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.355797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.27.1.193:34155)
0807 18:32:02.356510 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.356519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.357817 (+ 1298us) server_negotiation.cc:407] Connection header received
0807 18:32:02.357943 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.357950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.358160 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.358350 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:02.359091 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.360360 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.363211 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.364290 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.364386 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.364596 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.368149 (+ 3553us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.368275 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":376,"threads_started":1}
I20260807 18:32:02.370914 12908 tablet_service.cc:1467] Processing CreateTablet for tablet 970d87d0bb4043ce892eda6d2c973607 (DEFAULT_TABLE table=test-table [id=784d1c5b939f4fc9bf20dd2cf57cb3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:02.370918 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 970d87d0bb4043ce892eda6d2c973607 (DEFAULT_TABLE table=test-table [id=784d1c5b939f4fc9bf20dd2cf57cb3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:02.371261 12632 tablet_service.cc:1467] Processing CreateTablet for tablet 970d87d0bb4043ce892eda6d2c973607 (DEFAULT_TABLE table=test-table [id=784d1c5b939f4fc9bf20dd2cf57cb3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:02.372476 12908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970d87d0bb4043ce892eda6d2c973607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:02.372478 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970d87d0bb4043ce892eda6d2c973607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:02.372490 12632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970d87d0bb4043ce892eda6d2c973607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:02.385098 12992 tablet_bootstrap.cc:492] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Bootstrap starting.
I20260807 18:32:02.387815 12993 tablet_bootstrap.cc:492] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: Bootstrap starting.
I20260807 18:32:02.389164 12992 tablet_bootstrap.cc:654] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:02.389571 12994 tablet_bootstrap.cc:492] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: Bootstrap starting.
I20260807 18:32:02.391062 12992 log.cc:824] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:02.391508 12993 tablet_bootstrap.cc:654] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:02.392609 12994 tablet_bootstrap.cc:654] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:02.392639 12993 log.cc:824] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:02.393774 12994 log.cc:824] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:02.395334 12992 tablet_bootstrap.cc:492] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: No bootstrap required, opened a new log
I20260807 18:32:02.395617 12992 ts_tablet_manager.cc:1397] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260807 18:32:02.395862 12993 tablet_bootstrap.cc:492] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: No bootstrap required, opened a new log
I20260807 18:32:02.396131 12993 ts_tablet_manager.cc:1397] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 18:32:02.401274 12994 tablet_bootstrap.cc:492] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: No bootstrap required, opened a new log
I20260807 18:32:02.401543 12994 ts_tablet_manager.cc:1397] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260807 18:32:02.406672 12993 raft_consensus.cc:348] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:02.406674 12992 raft_consensus.cc:348] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:02.407114 12992 raft_consensus.cc:374] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:02.407234 12992 raft_consensus.cc:729] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebad1a5083d4be89c6824b27908118a, State: Initialized, Role: FOLLOWER
I20260807 18:32:02.407107 12993 raft_consensus.cc:374] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:02.407495 12993 raft_consensus.cc:729] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbf54c3d6184635acaeaf7ca5338792, State: Initialized, Role: FOLLOWER
I20260807 18:32:02.408041 12993 consensus_queue.cc:260] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:02.408017 12992 consensus_queue.cc:260] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:02.408802 12994 raft_consensus.cc:348] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:02.409237 12994 raft_consensus.cc:374] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:02.409343 12994 raft_consensus.cc:729] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0835388faabb4e5fb79613d6be2a85b6, State: Initialized, Role: FOLLOWER
I20260807 18:32:02.410159 12994 consensus_queue.cc:260] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:02.410962 12992 ts_tablet_manager.cc:1428] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260807 18:32:02.412151 12837 heartbeater.cc:503] Master 127.27.1.254:38497 was elected leader, sending a full tablet report...
I20260807 18:32:02.413386 12994 ts_tablet_manager.cc:1428] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260807 18:32:02.413532 12993 ts_tablet_manager.cc:1428] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: Time spent starting tablet: real 0.017s user 0.004s sys 0.012s
I20260807 18:32:02.413951 12976 heartbeater.cc:503] Master 127.27.1.254:38497 was elected leader, sending a full tablet report...
W20260807 18:32:02.501164 12977 tablet.cc:2369] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:32:02.528301 12838 tablet.cc:2369] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:32:02.566397 12699 tablet.cc:2369] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:32:02.575793 13002 raft_consensus.cc:482] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:32:02.576131 13002 raft_consensus.cc:504] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:02.577905 13002 leader_election.cc:288] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017), aebad1a5083d4be89c6824b27908118a (127.27.1.193:34155)
I20260807 18:32:02.584761 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.578352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:38017 (local address 127.27.1.195:57345)
0807 18:32:02.578760 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.578777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:02.578937 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:02.579366 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:02.579371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:02.579408 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:02.579637 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.579646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.580829 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.580834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.582771 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.582785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.584232 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.584236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.584363 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.584388 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 18:32:02.584463 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 18:32:02.584554 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"spinlock_wait_cycles":23424}
I20260807 18:32:02.584919 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.578468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:57345 (local address 127.27.1.194:38017)
0807 18:32:02.578693 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.578700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.578869 (+ 169us) server_negotiation.cc:407] Connection header received
0807 18:32:02.579110 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.579115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.579237 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.579336 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:02.579780 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.580686 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.582924 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.584094 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.584241 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.584321 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.584513 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.584643 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 18:32:02.585776 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.578477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:39437 (local address 127.27.1.193:34155)
0807 18:32:02.578693 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.578701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.579394 (+ 693us) server_negotiation.cc:407] Connection header received
0807 18:32:02.579601 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.579613 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.579671 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.579770 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:02.580380 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.581457 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.583602 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.584856 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.585019 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.585099 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.585520 (+ 421us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.585613 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:32:02.585978 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.578449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.27.1.195:39437)
0807 18:32:02.579286 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.579301 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:02.579458 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:02.579786 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:02.579791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:02.579814 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:02.580191 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.580204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.581602 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.581608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.583449 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:02.583461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.585189 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.585196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:02.585325 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.585379 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 18:32:02.585476 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 18:32:02.585614 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":2,"thread_start_us":491,"threads_started":1}
I20260807 18:32:02.585950 12791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970d87d0bb4043ce892eda6d2c973607" candidate_uuid: "0835388faabb4e5fb79613d6be2a85b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbf54c3d6184635acaeaf7ca5338792" is_pre_election: true
I20260807 18:32:02.586534 12791 raft_consensus.cc:2454] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0835388faabb4e5fb79613d6be2a85b6 in term 0.
I20260807 18:32:02.587035 12652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970d87d0bb4043ce892eda6d2c973607" candidate_uuid: "0835388faabb4e5fb79613d6be2a85b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aebad1a5083d4be89c6824b27908118a" is_pre_election: true
I20260807 18:32:02.587261 12861 leader_election.cc:302] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0835388faabb4e5fb79613d6be2a85b6, ddbf54c3d6184635acaeaf7ca5338792; no voters:
I20260807 18:32:02.587637 13002 raft_consensus.cc:2781] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:32:02.587635 12652 raft_consensus.cc:2454] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0835388faabb4e5fb79613d6be2a85b6 in term 0.
I20260807 18:32:02.587802 13002 raft_consensus.cc:482] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:32:02.587930 13002 raft_consensus.cc:3037] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:02.590207 13002 raft_consensus.cc:504] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:02.590797 13002 leader_election.cc:288] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [CANDIDATE]: Term 1 election: Requested vote from peers ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017), aebad1a5083d4be89c6824b27908118a (127.27.1.193:34155)
I20260807 18:32:02.591269 12791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970d87d0bb4043ce892eda6d2c973607" candidate_uuid: "0835388faabb4e5fb79613d6be2a85b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbf54c3d6184635acaeaf7ca5338792"
I20260807 18:32:02.591325 12652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970d87d0bb4043ce892eda6d2c973607" candidate_uuid: "0835388faabb4e5fb79613d6be2a85b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aebad1a5083d4be89c6824b27908118a"
I20260807 18:32:02.591447 12791 raft_consensus.cc:3037] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:02.591588 12652 raft_consensus.cc:3037] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:02.593688 12791 raft_consensus.cc:2454] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0835388faabb4e5fb79613d6be2a85b6 in term 1.
I20260807 18:32:02.593739 12652 raft_consensus.cc:2454] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0835388faabb4e5fb79613d6be2a85b6 in term 1.
I20260807 18:32:02.594029 12861 leader_election.cc:302] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0835388faabb4e5fb79613d6be2a85b6, ddbf54c3d6184635acaeaf7ca5338792; no voters:
I20260807 18:32:02.594326 13002 raft_consensus.cc:2781] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:32:02.594774 13002 raft_consensus.cc:686] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0835388faabb4e5fb79613d6be2a85b6, State: Running, Role: LEADER
I20260807 18:32:02.595302 13002 consensus_queue.cc:237] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:02.600037 12502 catalog_manager.cc:5495] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195). New cstate: current_term: 1 leader_uuid: "0835388faabb4e5fb79613d6be2a85b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } health_report { overall_health: UNKNOWN } } }
I20260807 18:32:02.650523 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.645420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.27.1.195:45379)
0807 18:32:02.645615 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.645620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.645728 (+ 108us) server_negotiation.cc:407] Connection header received
0807 18:32:02.645932 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.645935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.646028 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.646153 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:32:02.646470 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.647273 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.648548 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.648925 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.649012 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.649476 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:32:02.650073 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:32:02.650184 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.650261 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.650323 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 18:32:02.659564 12791 raft_consensus.cc:1261] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792 [term 1 FOLLOWER]: Refusing update from remote peer 0835388faabb4e5fb79613d6be2a85b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:32:02.659578 12652 raft_consensus.cc:1261] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Refusing update from remote peer 0835388faabb4e5fb79613d6be2a85b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:32:02.660533 13004 consensus_queue.cc:1035] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:32:02.660905 13002 consensus_queue.cc:1035] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:32:02.674383 13007 mvcc.cc:205] Tried to move back new op lower bound from 7315978332803072000 to 7315978332554153984. Current Snapshot: MvccSnapshot[applied={T|T < 7315978332803072000}]
I20260807 18:32:02.688223 12504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:32:02.688512 12504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:32:02.691789 12504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Sending DeleteTablet for 3 replicas of tablet 970d87d0bb4043ce892eda6d2c973607
I20260807 18:32:02.692873 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 970d87d0bb4043ce892eda6d2c973607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:02 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:59818
I20260807 18:32:02.693048 12908 tablet_service.cc:1514] Processing DeleteTablet for tablet 970d87d0bb4043ce892eda6d2c973607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:02 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:55466
I20260807 18:32:02.693333 12632 tablet_service.cc:1514] Processing DeleteTablet for tablet 970d87d0bb4043ce892eda6d2c973607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:02 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:51960
I20260807 18:32:02.694543 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 12542
I20260807 18:32:02.712412 13019 tablet_replica.cc:330] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: stopping tablet replica
I20260807 18:32:02.713398 13019 raft_consensus.cc:2229] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:32:02.715287 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.695360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.27.1.193:34155)
0807 18:32:02.695529 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.695535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.696073 (+ 538us) server_negotiation.cc:407] Connection header received
0807 18:32:02.696299 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.696305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.696403 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.696507 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:02.698781 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.699989 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.702373 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.703046 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.703193 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.707653 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:02.707668 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:02.707688 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:02.707735 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:02.711902 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:02.712889 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:02.712898 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:02.712901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:02.713057 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:02.713506 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:02.713511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:02.713515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:02.713991 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:02.714279 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.714633 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.714865 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 18:32:02.716336 12693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:02.717021 13018 tablet_replica.cc:330] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: stopping tablet replica
I20260807 18:32:02.717463 13018 raft_consensus.cc:2229] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:32:02.718202 13018 raft_consensus.cc:2258] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:02.720615 13018 ts_tablet_manager.cc:1905] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:02.725637 13017 tablet_replica.cc:330] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: stopping tablet replica
I20260807 18:32:02.726516 13017 raft_consensus.cc:2229] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:32:02.727222 13017 raft_consensus.cc:2258] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:02.728644 13018 ts_tablet_manager.cc:1918] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:02.728828 13018 log.cc:1196] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/970d87d0bb4043ce892eda6d2c973607
I20260807 18:32:02.729382 13018 ts_tablet_manager.cc:1939] T 970d87d0bb4043ce892eda6d2c973607 P 0835388faabb4e5fb79613d6be2a85b6: Deleting consensus metadata
I20260807 18:32:02.730007 13017 ts_tablet_manager.cc:1905] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:02.731285 12490 catalog_manager.cc:4841] TS 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379): tablet 970d87d0bb4043ce892eda6d2c973607 (table test-table [id=784d1c5b939f4fc9bf20dd2cf57cb3a7]) successfully deleted
I20260807 18:32:02.738451 13017 ts_tablet_manager.cc:1918] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:02.738646 13017 log.cc:1196] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/970d87d0bb4043ce892eda6d2c973607
I20260807 18:32:02.739184 13017 ts_tablet_manager.cc:1939] T 970d87d0bb4043ce892eda6d2c973607 P ddbf54c3d6184635acaeaf7ca5338792: Deleting consensus metadata
I20260807 18:32:02.740906 12489 catalog_manager.cc:4841] TS ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017): tablet 970d87d0bb4043ce892eda6d2c973607 (table test-table [id=784d1c5b939f4fc9bf20dd2cf57cb3a7]) successfully deleted
I20260807 18:32:02.857878 13019 raft_consensus.cc:2258] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:02.858595 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 12542
I20260807 18:32:02.880193 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 12702
W20260807 18:32:02.880151 12490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.1.193:34155 (error 108)
I20260807 18:32:02.882578 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.881719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.0.0.1:51978)
0807 18:32:02.882216 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.882351 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:34155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":271,"threads_started":1}
W20260807 18:32:02.882776 12490 catalog_manager.cc:4584] TS aebad1a5083d4be89c6824b27908118a (127.27.1.193:34155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 970d87d0bb4043ce892eda6d2c973607: Network error: Client connection negotiation failed: client connection to 127.27.1.193:34155: connect: Connection refused (error 111)
I20260807 18:32:02.894712 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.881310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.27.1.194:38017)
0807 18:32:02.881481 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:02.881485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:02.881503 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:32:02.881558 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:02.881561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:02.881636 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:02.881760 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:02.883589 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.884364 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.885616 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:02.885998 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:02.886139 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:02.889004 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:02.889017 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:02.889031 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:02.889066 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:02.891940 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:02.892652 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:02.892659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:02.892661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:02.892788 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:02.893179 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:02.893184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:02.893187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:02.893560 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:02.893856 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:02.894169 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 18:32:02.894382 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 18:32:02.895608 12832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:02.936698 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.936150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.0.0.1:51980)
0807 18:32:02.936404 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.936523 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:34155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 18:32:02.937948 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.937494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.0.0.1:51984)
0807 18:32:02.937676 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.937770 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:34155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 18:32:02.987763 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 12702
I20260807 18:32:02.998760 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.998122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.0.0.1:51994)
0807 18:32:02.998430 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.998553 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:34155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260807 18:32:03.000098 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:02.999632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.0.0.1:52006)
0807 18:32:02.999812 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:02.999915 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:34155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 18:32:03.009406 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 12840
I20260807 18:32:03.025775 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:03.010284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.27.1.195:45379)
0807 18:32:03.010486 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:03.010493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:03.010518 (+ 25us) server_negotiation.cc:407] Connection header received
0807 18:32:03.010648 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:03.010654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:03.010760 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:03.010872 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:03.012353 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:03.013564 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:03.014852 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:03.015575 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:03.015720 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:03.018375 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:03.018393 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:03.018415 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:03.018466 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:03.021819 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:03.022564 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:03.022572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:03.022574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:03.022780 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:03.023546 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:03.023553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:03.023556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:03.024051 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:03.024405 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:03.025321 (+ 916us) server_negotiation.cc:299] Negotiation successful
0807 18:32:03.025484 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 18:32:03.026703 12971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:03.081043 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:03.080525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.0.0.1:52020)
0807 18:32:03.080769 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:03.080870 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:34155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 18:32:03.082180 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:03.081786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.0.0.1:52030)
0807 18:32:03.081977 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:03.082047 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:34155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260807 18:32:03.152259 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 12840
I20260807 18:32:03.175200 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 12471
I20260807 18:32:03.176380 12535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:03.291911 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:03.291231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:34155 (local address 127.0.0.1:52044)
0807 18:32:03.291599 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:03.291724 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:34155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260807 18:32:03.292003 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 12471
I20260807 18:32:23.318368 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38497
--webserver_interface=127.27.1.254
--webserver_port=33951
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:38497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:23.663314 13026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:23.663636 13026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:23.663818 13026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:23.670842 13026 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 18:32:23.670977 13026 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 18:32:23.671072 13026 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 18:32:23.671197 13026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:32:23.671334 13026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:23.671434 13026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:23.671518 13026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:32:23.671597 13026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:32:23.681329 13026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:38497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=33951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:23.682643 13026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:23.684028 13026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:23.696329 13031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:23.696281 13032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:23.696707 13026 server_base.cc:1034] running on GCE node
W20260807 18:32:23.696386 13034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:23.697631 13026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:23.699062 13026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:23.700325 13026 hybrid_clock.cc:648] HybridClock initialized: now 1786127543700236 us; error 59 us; skew 500 ppm
I20260807 18:32:23.700651 13026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:23 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127543, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:23.703830 13026 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:32:23.705192 13026 webserver.cc:460] Webserver started at http://127.27.1.254:33951/ using document root <none> and password file <none>
I20260807 18:32:23.705864 13026 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:23.705991 13026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:23.711161 13026 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:32:23.713943 13041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:23.715196 13026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 18:32:23.715366 13026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "ef766a57049e42e9a58120e94356a8fb"
format_stamp: "Formatted at 2026-08-07 18:31:37 on dist-test-slave-1xrh"
server_key: "7b1f66883245590bcd26511e47ab65ae"
server_key_iv: "d8946b390ed258936f2cd19fa316e606"
server_key_version: "encryptionkey@0"
I20260807 18:32:23.716286 13026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:23.748063 13026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:23.752856 13026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:23.753118 13026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:23.771099 13026 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:38497
I20260807 18:32:23.771170 13095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:38497 every 8 connection(s)
I20260807 18:32:23.772684 13026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:32:23.779793 13096 sys_catalog.cc:263] Verifying existing consensus state
I20260807 18:32:23.779919 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 13026
I20260807 18:32:23.780933 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:34155
--local_ip_for_outbound_sockets=127.27.1.193
--tserver_master_addrs=127.27.1.254:38497
--webserver_port=40463
--webserver_interface=127.27.1.193
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:32:23.784940 13096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Bootstrap starting.
I20260807 18:32:23.807967 13096 log.cc:824] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:23.824175 13096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 18:32:23.824712 13096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Bootstrap complete.
I20260807 18:32:23.833729 13096 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:32:23.834592 13096 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef766a57049e42e9a58120e94356a8fb, State: Initialized, Role: FOLLOWER
I20260807 18:32:23.835279 13096 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:32:23.835543 13096 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:32:23.835667 13096 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:32:23.835855 13096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 2 FOLLOWER]: Advancing to term 3
I20260807 18:32:23.839664 13096 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:32:23.840134 13096 leader_election.cc:302] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef766a57049e42e9a58120e94356a8fb; no voters:
I20260807 18:32:23.840823 13096 leader_election.cc:288] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 18:32:23.840955 13100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 3 FOLLOWER]: Leader election won for term 3
I20260807 18:32:23.842226 13100 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [term 3 LEADER]: Becoming Leader. State: Replica: ef766a57049e42e9a58120e94356a8fb, State: Running, Role: LEADER
I20260807 18:32:23.842841 13100 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } }
I20260807 18:32:23.843094 13096 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:32:23.845526 13102 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef766a57049e42e9a58120e94356a8fb. Latest consensus state: current_term: 3 leader_uuid: "ef766a57049e42e9a58120e94356a8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } } }
I20260807 18:32:23.845935 13102 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:23.845640 13101 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ef766a57049e42e9a58120e94356a8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef766a57049e42e9a58120e94356a8fb" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38497 } } }
I20260807 18:32:23.846086 13101 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:23.847308 13107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:32:23.862607 13107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:32:23.864671 13107 catalog_manager.cc:1259] Loaded cluster ID: 8de8184198ed4def902a4b9e3f65724e
I20260807 18:32:23.864878 13107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:32:23.870633 13107 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:32:23.872491 13107 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Loaded TSK: 0
I20260807 18:32:23.872736 13107 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef766a57049e42e9a58120e94356a8fb: Loaded TSK: 1
F20260807 18:32:23.873916 13107 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7ba307a9ad google::LogMessage::Fail() at ??:0
@ 0x7f7ba307d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7ba307a4de google::LogMessage::Flush() at ??:0
@ 0x7f7ba307ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7ba6dde7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7ba6dcd73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7bb8aa91c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
I20260807 18:32:23.900164 13118 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f7bb8aa8783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7bb8b342a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7bb8b340c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7bb8b7e5e9 std::function<>::operator()() at ??:0
@ 0x7f7bb8aaccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7bb8aaac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7bb8b32610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7bb8b32439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7ba5cc6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7ba5cc991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7ba5cc9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7ba5ca4eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7ba8f7e6db start_thread at ??:0
@ 0x7f7ba1f1461f clone at ??:0
W20260807 18:32:24.168265 13098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:24.168586 13098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:24.168759 13098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:24.176491 13098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:24.176640 13098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:24.176823 13098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:32:24.186620 13098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:34155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=40463
--tserver_master_addrs=127.27.1.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:24.187933 13098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:24.189235 13098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:24.202395 13144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:24.202440 13147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:24.202548 13145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:24.203477 13098 server_base.cc:1034] running on GCE node
I20260807 18:32:24.204073 13098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:24.204993 13098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:24.206229 13098 hybrid_clock.cc:648] HybridClock initialized: now 1786127544206187 us; error 31 us; skew 500 ppm
I20260807 18:32:24.206558 13098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:24 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127544, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:24.209734 13098 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:32:24.211330 13098 webserver.cc:460] Webserver started at http://127.27.1.193:40463/ using document root <none> and password file <none>
I20260807 18:32:24.212013 13098 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:24.212172 13098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:24.217561 13098 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 18:32:24.220377 13154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:24.221616 13098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:32:24.221784 13098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "aebad1a5083d4be89c6824b27908118a"
format_stamp: "Formatted at 2026-08-07 18:31:38 on dist-test-slave-1xrh"
server_key: "aa67e8f436a8e79f369c3fb899ce26d5"
server_key_iv: "32a50c8b5c67d6beb95982d437dfcd3c"
server_key_version: "encryptionkey@0"
I20260807 18:32:24.222740 13098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:24.239061 13098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:24.242808 13098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:24.243093 13098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:24.244104 13098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:24.245857 13161 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 18:32:24.251428 13098 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 18:32:24.251572 13098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260807 18:32:24.251716 13098 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 18:32:24.254823 13098 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 18:32:24.254940 13098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260807 18:32:24.255393 13161 tablet_bootstrap.cc:492] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Bootstrap starting.
I20260807 18:32:24.282797 13161 log.cc:824] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:24.295463 13098 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:34155
I20260807 18:32:24.295512 13272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:34155 every 8 connection(s)
I20260807 18:32:24.296142 13161 tablet_bootstrap.cc:492] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 18:32:24.296689 13161 tablet_bootstrap.cc:492] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Bootstrap complete.
I20260807 18:32:24.296794 13098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:32:24.298274 13161 ts_tablet_manager.cc:1397] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Time spent bootstrapping tablet: real 0.043s user 0.018s sys 0.021s
I20260807 18:32:24.302246 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:24.299839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:34601)
0807 18:32:24.300762 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:24.301069 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":412,"threads_started":1}
W20260807 18:32:24.303046 13273 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:38497 (0 consecutive failures): Network error: Failed to ping master at 127.27.1.254:38497: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
I20260807 18:32:24.303399 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 13098
I20260807 18:32:24.304190 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:38017
--local_ip_for_outbound_sockets=127.27.1.194
--tserver_master_addrs=127.27.1.254:38497
--webserver_port=38241
--webserver_interface=127.27.1.194
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:32:24.304752 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:24.303658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:48641)
0807 18:32:24.304567 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:24.304636 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":787}
I20260807 18:32:24.305680 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:24.305318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:57039)
0807 18:32:24.305464 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:24.305550 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 18:32:24.305931 13273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 18:32:24.308423 13161 raft_consensus.cc:348] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:24.309844 13161 raft_consensus.cc:729] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebad1a5083d4be89c6824b27908118a, State: Initialized, Role: FOLLOWER
I20260807 18:32:24.310987 13161 consensus_queue.cc:260] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:24.312888 13161 ts_tablet_manager.cc:1428] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260807 18:32:24.312984 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:24.312464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:52683)
0807 18:32:24.312717 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:24.312809 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260807 18:32:24.314343 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:24.313837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:60465)
0807 18:32:24.314069 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:24.314183 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260807 18:32:24.650076 13277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:24.650384 13277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:24.650554 13277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:24.657553 13277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:24.657704 13277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:24.657886 13277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:32:24.667568 13277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:38017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=38241
--tserver_master_addrs=127.27.1.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:24.668883 13277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:24.670217 13277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:24.683503 13285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:24.683624 13282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:24.683590 13283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:24.684652 13277 server_base.cc:1034] running on GCE node
I20260807 18:32:24.685230 13277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:24.686089 13277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:24.687294 13277 hybrid_clock.cc:648] HybridClock initialized: now 1786127544687250 us; error 53 us; skew 500 ppm
I20260807 18:32:24.687592 13277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:24 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127544, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:24.690691 13277 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:32:24.692119 13277 webserver.cc:460] Webserver started at http://127.27.1.194:38241/ using document root <none> and password file <none>
I20260807 18:32:24.692736 13277 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:24.692862 13277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:24.697917 13277 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 18:32:24.700526 13294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:24.701846 13277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 18:32:24.701978 13277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "ddbf54c3d6184635acaeaf7ca5338792"
format_stamp: "Formatted at 2026-08-07 18:31:38 on dist-test-slave-1xrh"
server_key: "c910a5c1d8387683266f04bce2701379"
server_key_iv: "a83c4af1dd393da9825dd02911338a17"
server_key_version: "encryptionkey@0"
I20260807 18:32:24.702888 13277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:24.718207 13277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:24.722287 13277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:24.722508 13277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:24.723459 13277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:24.724756 13277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:24.724845 13277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:24.724921 13277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:24.724978 13277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:24.759029 13277 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:38017
I20260807 18:32:24.759105 13413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:38017 every 8 connection(s)
I20260807 18:32:24.760516 13277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:32:24.765220 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:24.763429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:53745)
0807 18:32:24.764192 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:24.764393 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":283,"threads_started":1}
W20260807 18:32:24.765919 13414 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:38497 (0 consecutive failures): Network error: Failed to ping master at 127.27.1.254:38497: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
I20260807 18:32:24.766718 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:24.766417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:52893)
0807 18:32:24.766548 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:24.766620 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 18:32:24.767562 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:24.767268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:41181)
0807 18:32:24.767379 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:24.767453 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 18:32:24.767810 13414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 18:32:24.769665 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 13277
I20260807 18:32:24.770277 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:45379
--local_ip_for_outbound_sockets=127.27.1.195
--tserver_master_addrs=127.27.1.254:38497
--webserver_port=45303
--webserver_interface=127.27.1.195
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:25.110029 13417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:25.110303 13417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:25.110486 13417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:25.117697 13417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:25.117795 13417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:25.117954 13417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:32:25.127825 13417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:45379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=45303
--tserver_master_addrs=127.27.1.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:25.129091 13417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:25.130414 13417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:25.144129 13422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:25.144167 13423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:25.144467 13425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:25.145323 13417 server_base.cc:1034] running on GCE node
I20260807 18:32:25.145943 13417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:25.146872 13417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:25.148093 13417 hybrid_clock.cc:648] HybridClock initialized: now 1786127545148053 us; error 42 us; skew 500 ppm
I20260807 18:32:25.148408 13417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:25 dist-test-slave-1xrh krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127545, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:25.151438 13417 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:32:25.152822 13417 webserver.cc:460] Webserver started at http://127.27.1.195:45303/ using document root <none> and password file <none>
I20260807 18:32:25.153493 13417 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:25.153644 13417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:25.159018 13417 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260807 18:32:25.161716 13432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:25.162925 13417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:32:25.163108 13417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "0835388faabb4e5fb79613d6be2a85b6"
format_stamp: "Formatted at 2026-08-07 18:31:39 on dist-test-slave-1xrh"
server_key: "32a2c86021a2924ecd74bb2664b2dbfe"
server_key_iv: "c1f0986725af5c667b84b80601a7332d"
server_key_version: "encryptionkey@0"
I20260807 18:32:25.164134 13417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:25.177673 13417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:25.181488 13417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:25.181761 13417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:25.182724 13417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:25.184136 13417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:25.184268 13417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:25.184399 13417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:25.184487 13417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:25.218560 13417 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:45379
I20260807 18:32:25.218644 13548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:45379 every 8 connection(s)
I20260807 18:32:25.220310 13417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:32:25.225131 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:25.223238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:52567)
0807 18:32:25.224022 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:25.224225 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":331,"threads_started":1}
W20260807 18:32:25.225756 13549 heartbeater.cc:644] Failed to heartbeat to 127.27.1.254:38497 (0 consecutive failures): Network error: Failed to ping master at 127.27.1.254:38497: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
I20260807 18:32:25.226614 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:25.226260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:55087)
0807 18:32:25.226447 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:25.226515 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260807 18:32:25.227593 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:25.227211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:39525)
0807 18:32:25.227350 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:25.227480 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 18:32:25.227880 13549 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 18:32:25.229331 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 13417
I20260807 18:32:25.316234 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:25.315335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:43223)
0807 18:32:25.315920 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:25.316033 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":277,"threads_started":1}
I20260807 18:32:25.457376 13554 raft_consensus.cc:482] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:32:25.457688 13554 raft_consensus.cc:504] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:25.459448 13554 leader_election.cc:288] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017), 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379)
Aug 07 18:32:25 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127544, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
Aug 07 18:32:25 dist-test-slave-1xrh krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127544, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:32:25.481168 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:25.459851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:38017 (local address 127.27.1.193:57267)
0807 18:32:25.460201 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:25.460252 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:25.460921 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:25.464769 (+ 3848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:25.464782 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:25.465643 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:25.466355 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:25.466375 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:25.468120 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:25.468123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:25.469561 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:25.469573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:25.470594 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:25.470602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:25.470799 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:25.471650 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:25.471668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:25.474645 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:25.478189 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:25.478195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:25.478199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:25.478574 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:25.479104 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:25.479107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:25.479109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:25.479298 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:25.480081 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:25.480085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:25.480218 (+ 133us) client_negotiation.cc:769] Sending connection context
0807 18:32:25.480349 (+ 131us) client_negotiation.cc:240] Negotiation successful
0807 18:32:25.480675 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":85}
I20260807 18:32:25.481266 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:25.460148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:47495 (local address 127.27.1.195:45379)
0807 18:32:25.462633 (+ 2485us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:25.462647 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:25.462690 (+ 43us) server_negotiation.cc:407] Connection header received
0807 18:32:25.464297 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:25.464305 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:25.464679 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:25.465034 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:25.466559 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:25.467878 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:25.469703 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:25.470315 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:25.470529 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:25.474355 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:25.474366 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:25.474383 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:25.474435 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:25.477311 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:25.478092 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:25.478101 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:25.478108 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:25.478273 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:25.478765 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:25.478771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:25.478777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:25.479500 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:25.479682 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:25.480431 (+ 749us) server_negotiation.cc:299] Negotiation successful
0807 18:32:25.480862 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2305,"thread_start_us":2150,"threads_started":1}
I20260807 18:32:25.481341 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:25.459940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:57267 (local address 127.27.1.194:38017)
0807 18:32:25.462233 (+ 2293us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:25.462253 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:25.462303 (+ 50us) server_negotiation.cc:407] Connection header received
0807 18:32:25.463915 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:25.463925 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:25.464299 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:25.464564 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:25.466570 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:25.467931 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:25.469705 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:25.470317 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:25.470514 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:25.474876 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:25.474883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:25.474893 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:25.474928 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:25.478063 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:25.478698 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:25.478710 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:25.478730 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:25.478964 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:25.479409 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:25.479414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:25.479418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:25.479940 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:25.480076 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:25.480574 (+ 498us) server_negotiation.cc:299] Negotiation successful
0807 18:32:25.480914 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2056,"thread_start_us":1923,"threads_started":1}
I20260807 18:32:25.481209 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:25.460010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:45379 (local address 127.27.1.193:47495)
0807 18:32:25.461758 (+ 1748us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:25.461780 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:25.461904 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:25.465002 (+ 3098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:25.465012 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:25.465642 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:25.466360 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:25.466378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:25.468077 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:25.468085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:25.469558 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:25.469570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:25.470520 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:25.470527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:25.470795 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:25.471591 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:25.471615 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:25.474058 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:25.477527 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:25.477534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:25.477544 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:25.477936 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:25.478408 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:25.478412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:25.478414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:25.478596 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:25.479640 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:25.479646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:25.479906 (+ 260us) client_negotiation.cc:769] Sending connection context
0807 18:32:25.480202 (+ 296us) client_negotiation.cc:240] Negotiation successful
0807 18:32:25.480810 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1585,"mutex_wait_us":304,"spinlock_wait_cycles":25600,"thread_start_us":1396,"threads_started":1}
W20260807 18:32:25.485001 13157 leader_election.cc:341] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379): Not found: Tablet not found: 970d87d0bb4043ce892eda6d2c973607
W20260807 18:32:25.485265 13156 leader_election.cc:341] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017): Not found: Tablet not found: 970d87d0bb4043ce892eda6d2c973607
I20260807 18:32:25.485436 13156 leader_election.cc:302] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aebad1a5083d4be89c6824b27908118a; no voters: 0835388faabb4e5fb79613d6be2a85b6, ddbf54c3d6184635acaeaf7ca5338792
I20260807 18:32:25.485842 13554 raft_consensus.cc:2726] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:32:25.769269 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:25.768346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:38075)
0807 18:32:25.768867 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:25.769020 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":250,"threads_started":1}
I20260807 18:32:26.229157 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:26.228431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:59457)
0807 18:32:26.228828 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:26.228931 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":195,"threads_started":1}
I20260807 18:32:26.317679 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:26.316985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:42579)
0807 18:32:26.317434 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:26.317508 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":235,"threads_started":1}
I20260807 18:32:26.770723 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:26.770041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:44579)
0807 18:32:26.770465 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:26.770551 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":234,"threads_started":1}
I20260807 18:32:27.230650 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:27.229986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:36989)
0807 18:32:27.230409 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:27.230484 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260807 18:32:27.319660 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:27.318615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:45149)
0807 18:32:27.319272 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:27.319389 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":243,"threads_started":1}
I20260807 18:32:27.452868 13564 raft_consensus.cc:482] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:32:27.453116 13564 raft_consensus.cc:504] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:27.453893 13564 leader_election.cc:288] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017), 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379)
W20260807 18:32:27.455091 13156 leader_election.cc:341] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017): Not found: Tablet not found: 970d87d0bb4043ce892eda6d2c973607
W20260807 18:32:27.455271 13157 leader_election.cc:341] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379): Not found: Tablet not found: 970d87d0bb4043ce892eda6d2c973607
I20260807 18:32:27.455451 13157 leader_election.cc:302] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aebad1a5083d4be89c6824b27908118a; no voters: 0835388faabb4e5fb79613d6be2a85b6, ddbf54c3d6184635acaeaf7ca5338792
I20260807 18:32:27.455729 13564 raft_consensus.cc:2726] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:32:27.772342 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:27.771612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:34941)
0807 18:32:27.772027 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:27.772116 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":208,"threads_started":1}
I20260807 18:32:28.232321 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:28.231642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:49305)
0807 18:32:28.232077 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:28.232171 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":225,"threads_started":1}
I20260807 18:32:28.321614 13567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:28.320832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:59867)
0807 18:32:28.321312 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:28.321431 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":245,"threads_started":1}
I20260807 18:32:28.774039 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:28.773321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:59241)
0807 18:32:28.773769 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:28.773854 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":215,"threads_started":1}
I20260807 18:32:29.234001 13569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:29.233257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:48697)
0807 18:32:29.233711 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:29.233797 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":225,"threads_started":1}
I20260807 18:32:29.323562 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:29.322727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:39513)
0807 18:32:29.323252 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:29.323331 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":235,"threads_started":1}
I20260807 18:32:29.385468 13571 raft_consensus.cc:482] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:32:29.385711 13571 raft_consensus.cc:504] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:29.386452 13571 leader_election.cc:288] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017), 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379)
W20260807 18:32:29.387497 13156 leader_election.cc:341] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017): Not found: Tablet not found: 970d87d0bb4043ce892eda6d2c973607
W20260807 18:32:29.387656 13157 leader_election.cc:341] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379): Not found: Tablet not found: 970d87d0bb4043ce892eda6d2c973607
I20260807 18:32:29.387766 13157 leader_election.cc:302] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aebad1a5083d4be89c6824b27908118a; no voters: 0835388faabb4e5fb79613d6be2a85b6, ddbf54c3d6184635acaeaf7ca5338792
I20260807 18:32:29.388003 13571 raft_consensus.cc:2726] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:32:29.775947 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:29.774999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:33451)
0807 18:32:29.775611 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:29.775713 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":327,"threads_started":1}
I20260807 18:32:30.235649 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:30.234866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:50169)
0807 18:32:30.235407 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:30.235488 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":340,"threads_started":1}
I20260807 18:32:30.325232 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:30.324504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:48315)
0807 18:32:30.324958 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:30.325067 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":232,"threads_started":1}
I20260807 18:32:30.777633 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:30.776906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:52631)
0807 18:32:30.777356 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:30.777436 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":243,"threads_started":1}
I20260807 18:32:31.237250 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:31.236559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:32881)
0807 18:32:31.236997 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:31.237073 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":238,"threads_started":1}
I20260807 18:32:31.327190 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:31.326077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:44511)
0807 18:32:31.326705 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:31.326869 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
I20260807 18:32:31.779426 13578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:31.778590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:41383)
0807 18:32:31.779087 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:31.779227 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":210,"threads_started":1}
I20260807 18:32:32.238919 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:32.238153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:47547)
0807 18:32:32.238648 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:32.238719 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":216,"threads_started":1}
I20260807 18:32:32.328696 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:32.328030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:53189)
0807 18:32:32.328449 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:32.328540 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":211,"threads_started":1}
I20260807 18:32:32.781251 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:32.780387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:40537)
0807 18:32:32.780937 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:32.781037 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":306,"threads_started":1}
I20260807 18:32:33.240540 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:33.239834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:45551)
0807 18:32:33.240257 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:33.240335 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260807 18:32:33.330317 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:33.329536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:57153)
0807 18:32:33.330044 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:33.330125 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":247,"threads_started":1}
I20260807 18:32:33.783216 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:33.782372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:49665)
0807 18:32:33.782910 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:33.782991 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
I20260807 18:32:34.048276 13585 raft_consensus.cc:482] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:32:34.048483 13585 raft_consensus.cc:504] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:34.049170 13585 leader_election.cc:288] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017), 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379)
W20260807 18:32:34.050125 13156 leader_election.cc:341] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017): Not found: Tablet not found: 970d87d0bb4043ce892eda6d2c973607
W20260807 18:32:34.050297 13157 leader_election.cc:341] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379): Not found: Tablet not found: 970d87d0bb4043ce892eda6d2c973607
I20260807 18:32:34.050453 13157 leader_election.cc:302] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aebad1a5083d4be89c6824b27908118a; no voters: 0835388faabb4e5fb79613d6be2a85b6, ddbf54c3d6184635acaeaf7ca5338792
I20260807 18:32:34.050714 13585 raft_consensus.cc:2726] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:32:34.242136 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:34.241400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:33705)
0807 18:32:34.241861 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:34.241955 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260807 18:32:34.332072 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:34.331309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:42885)
0807 18:32:34.331770 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:34.331858 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":233,"threads_started":1}
I20260807 18:32:34.784934 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:34.784132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:56861)
0807 18:32:34.784640 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:34.784737 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":276,"threads_started":1}
I20260807 18:32:35.243818 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:35.243018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:46165)
0807 18:32:35.243517 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:35.243615 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":295,"threads_started":1}
I20260807 18:32:35.333760 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:35.333040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:40073)
0807 18:32:35.333469 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:35.333568 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":220,"threads_started":1}
I20260807 18:32:35.786613 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:35.785878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:40117)
0807 18:32:35.786350 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:35.786434 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":247,"threads_started":1}
I20260807 18:32:36.245568 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:36.244736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:45257)
0807 18:32:36.245301 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:36.245398 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":268,"threads_started":1}
I20260807 18:32:36.335454 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:36.334579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:44139)
0807 18:32:36.335008 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:36.335101 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":230,"threads_started":1}
I20260807 18:32:36.788406 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:36.787580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:36343)
0807 18:32:36.788124 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:36.788228 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":300,"threads_started":1}
I20260807 18:32:37.247257 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:37.246480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:57475)
0807 18:32:37.246944 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:37.247023 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":255,"threads_started":1}
I20260807 18:32:37.337054 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:37.336277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:50925)
0807 18:32:37.336762 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:37.336844 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":266,"threads_started":1}
I20260807 18:32:37.790097 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:37.789334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:57349)
0807 18:32:37.789832 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:37.789923 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":264,"threads_started":1}
I20260807 18:32:38.249046 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:38.248162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:46633)
0807 18:32:38.248706 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:38.248848 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":322,"threads_started":1}
I20260807 18:32:38.338716 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:38.337970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:48783)
0807 18:32:38.338427 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:38.338511 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":229,"threads_started":1}
I20260807 18:32:38.792073 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:38.791045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:34029)
0807 18:32:38.791695 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:38.791787 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":375,"threads_started":1}
I20260807 18:32:39.250756 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:39.250012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:38099)
0807 18:32:39.250456 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:39.250553 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260807 18:32:39.340812 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:39.340011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:42453)
0807 18:32:39.340517 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:39.340626 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
I20260807 18:32:39.569813 13606 raft_consensus.cc:482] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:32:39.570045 13606 raft_consensus.cc:504] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf54c3d6184635acaeaf7ca5338792" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 38017 } } peers { permanent_uuid: "0835388faabb4e5fb79613d6be2a85b6" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 45379 } } peers { permanent_uuid: "aebad1a5083d4be89c6824b27908118a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 34155 } }
I20260807 18:32:39.570746 13606 leader_election.cc:288] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017), 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379)
W20260807 18:32:39.571883 13156 leader_election.cc:341] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ddbf54c3d6184635acaeaf7ca5338792 (127.27.1.194:38017): Not found: Tablet not found: 970d87d0bb4043ce892eda6d2c973607
W20260807 18:32:39.572099 13157 leader_election.cc:341] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0835388faabb4e5fb79613d6be2a85b6 (127.27.1.195:45379): Not found: Tablet not found: 970d87d0bb4043ce892eda6d2c973607
I20260807 18:32:39.572285 13157 leader_election.cc:302] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aebad1a5083d4be89c6824b27908118a; no voters: 0835388faabb4e5fb79613d6be2a85b6, ddbf54c3d6184635acaeaf7ca5338792
I20260807 18:32:39.572557 13606 raft_consensus.cc:2726] T 970d87d0bb4043ce892eda6d2c973607 P aebad1a5083d4be89c6824b27908118a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:32:39.794111 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:39.793359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.194:42317)
0807 18:32:39.793857 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:39.793944 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":272,"threads_started":1}
I20260807 18:32:40.235827 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 13098
I20260807 18:32:40.252785 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:40.251905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.195:53547)
0807 18:32:40.252405 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:40.252532 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":252,"threads_started":1}
I20260807 18:32:40.260286 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:40.236853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.27.1.193:34155)
0807 18:32:40.237398 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:40.237405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:40.237432 (+ 27us) server_negotiation.cc:407] Connection header received
0807 18:32:40.237536 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:40.237543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:40.237881 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:40.238061 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:40.240106 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:40.241073 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:40.242287 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:40.242859 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:40.242956 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:40.254606 (+ 11650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:40.254620 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:40.254636 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:40.254674 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:40.257651 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:40.258285 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:40.258292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:40.258294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:40.258429 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:40.258759 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:40.258763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:40.258764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:40.259186 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:40.259440 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:40.259707 (+ 267us) server_negotiation.cc:299] Negotiation successful
0807 18:32:40.259944 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
I20260807 18:32:40.261637 13268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:40.357137 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:40.350980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38497 (local address 127.27.1.193:58309)
0807 18:32:40.356680 (+ 5700us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:40.356832 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5569,"thread_start_us":5389,"threads_started":1}
I20260807 18:32:40.357537 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 13098
I20260807 18:32:40.379863 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 13277
I20260807 18:32:40.393364 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:40.380688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.27.1.194:38017)
0807 18:32:40.381289 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:40.381294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:40.381311 (+ 17us) server_negotiation.cc:407] Connection header received
0807 18:32:40.381378 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:40.381382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:40.381444 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:40.381541 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:40.382872 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:40.383761 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:40.384994 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:40.385436 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:40.385535 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:40.388159 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:40.388164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:40.388167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:40.388200 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:40.390987 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:40.391658 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:40.391662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:40.391664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:40.391794 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:40.392106 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:40.392109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:40.392111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:40.392366 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:40.392584 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:40.392782 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 18:32:40.392943 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":351,"threads_started":1}
I20260807 18:32:40.394081 13409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:40.479859 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 13277
I20260807 18:32:40.501725 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 13417
I20260807 18:32:40.516191 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:40.502724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.27.1.195:45379)
0807 18:32:40.503499 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:40.503504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:40.503523 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:32:40.503581 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:40.503585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:40.503656 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:40.503747 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:40.505211 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:40.506018 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:40.507441 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:40.507891 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:40.507992 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:40.510680 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:40.510684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:40.510688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:40.510722 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:40.513526 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:40.514255 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:40.514259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:40.514261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:40.514393 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:40.514723 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:40.514726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:40.514728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:40.514987 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:40.515264 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:40.515581 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 18:32:40.515748 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":438,"threads_started":1}
I20260807 18:32:40.516916 13544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:40.600817 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 13417
2026-08-07T18:32:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64881 ms)
[----------] 30 tests from SecurityITest (305543 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:32:40 dist-test-slave-1xrh krb5kdc[13619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:32:40 dist-test-slave-1xrh krb5kdc[13619](info): set up 2 sockets
Aug 07 18:32:40 dist-test-slave-1xrh krb5kdc[13619](info): commencing operation
krb5kdc: starting...
W20260807 18:32:41.830971 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:32:42 dist-test-slave-1xrh krb5kdc[13619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:32:42.140224 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34163
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:34163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:42.488574 13633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:42.489053 13633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:42.489233 13633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:42.499214 13633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:32:42.499348 13633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:42.499434 13633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:42.499524 13633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:32:42.499591 13633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:32:42.518234 13633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:34163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:42.520274 13633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:42.522238 13633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:42.536008 13639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:42.535966 13640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:42.536047 13643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:42.537698 13633 server_base.cc:1034] running on GCE node
I20260807 18:32:42.538496 13633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:42.539948 13633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:42.541186 13633 hybrid_clock.cc:648] HybridClock initialized: now 1786127562541151 us; error 51 us; skew 500 ppm
I20260807 18:32:42.541544 13633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:42 dist-test-slave-1xrh krb5kdc[13619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127562, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:42.544893 13633 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:32:42.546455 13633 webserver.cc:460] Webserver started at http://127.27.1.254:36375/ using document root <none> and password file <none>
I20260807 18:32:42.547222 13633 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:42.547374 13633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:42.547700 13633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:42.550695 13633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "62e2d60374554e94b14978978c34529e"
format_stamp: "Formatted at 2026-08-07 18:32:42 on dist-test-slave-1xrh"
server_key: "c3ce9db43ca2a6b401f254b788798fb7"
server_key_iv: "71c6037b13e74b8ea060b8adf7652e52"
server_key_version: "encryptionkey@0"
I20260807 18:32:42.551530 13633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "62e2d60374554e94b14978978c34529e"
format_stamp: "Formatted at 2026-08-07 18:32:42 on dist-test-slave-1xrh"
server_key: "c3ce9db43ca2a6b401f254b788798fb7"
server_key_iv: "71c6037b13e74b8ea060b8adf7652e52"
server_key_version: "encryptionkey@0"
I20260807 18:32:42.556718 13633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:32:42.560189 13650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:42.561446 13633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:32:42.561646 13633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "62e2d60374554e94b14978978c34529e"
format_stamp: "Formatted at 2026-08-07 18:32:42 on dist-test-slave-1xrh"
server_key: "c3ce9db43ca2a6b401f254b788798fb7"
server_key_iv: "71c6037b13e74b8ea060b8adf7652e52"
server_key_version: "encryptionkey@0"
I20260807 18:32:42.561851 13633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:42.586925 13633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:42.591015 13633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:42.591323 13633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:42.610299 13633 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:34163
I20260807 18:32:42.610333 13703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:34163 every 8 connection(s)
I20260807 18:32:42.611646 13633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:32:42.615443 13704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:42.621378 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 13633
I20260807 18:32:42.621657 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:32:42.622551 27655 external_mini_cluster.cc:1428] Setting key e9e4b79e16888c9e2bd87e9da253a59d
I20260807 18:32:42.626891 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e: Bootstrap starting.
I20260807 18:32:42.630703 13704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:42.631944 13704 log.cc:824] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:42.635094 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e: No bootstrap required, opened a new log
Aug 07 18:32:42 dist-test-slave-1xrh krb5kdc[13619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:42.643366 13704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2d60374554e94b14978978c34529e" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34163 } }
I20260807 18:32:42.643766 13704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:42.643910 13704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e2d60374554e94b14978978c34529e, State: Initialized, Role: FOLLOWER
I20260807 18:32:42.644536 13704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2d60374554e94b14978978c34529e" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34163 } }
I20260807 18:32:42.644814 13704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:32:42.644968 13704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:32:42.645180 13704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:42.646371 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:42.624756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.27.1.254:34163)
0807 18:32:42.625560 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:42.625572 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:42.625613 (+ 41us) server_negotiation.cc:407] Connection header received
0807 18:32:42.627368 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:42.627391 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:42.627738 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:42.628052 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:42.629991 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:42.631393 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:42.633141 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:42.633786 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:42.633916 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:42.638777 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:42.638785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:42.638801 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:42.638837 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:42.642014 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:42.642718 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:42.642726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:42.642734 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:42.642897 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:42.643370 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:42.643375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:42.643383 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:42.644045 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:42.644363 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:42.644836 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 18:32:42.645298 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":394,"threads_started":1}
I20260807 18:32:42.647783 13704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2d60374554e94b14978978c34529e" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34163 } }
I20260807 18:32:42.648277 13704 leader_election.cc:302] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62e2d60374554e94b14978978c34529e; no voters:
I20260807 18:32:42.649016 13704 leader_election.cc:288] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:32:42.649122 13709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:32:42.649835 13709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [term 1 LEADER]: Becoming Leader. State: Replica: 62e2d60374554e94b14978978c34529e, State: Running, Role: LEADER
I20260807 18:32:42.650439 13709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2d60374554e94b14978978c34529e" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34163 } }
I20260807 18:32:42.651320 13704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:32:42.655300 13710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62e2d60374554e94b14978978c34529e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2d60374554e94b14978978c34529e" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34163 } } }
I20260807 18:32:42.655776 13710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:42.655653 13712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62e2d60374554e94b14978978c34529e. Latest consensus state: current_term: 1 leader_uuid: "62e2d60374554e94b14978978c34529e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2d60374554e94b14978978c34529e" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34163 } } }
I20260807 18:32:42.656077 13712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:42.656702 13719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:32:42.661700 13719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:32:42.670719 13719 catalog_manager.cc:1347] Generated new cluster ID: 830be0308eb84962a302a901bcf910a4
I20260807 18:32:42.670855 13719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:32:42.696439 13719 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:32:42.697814 13719 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:32:42.710054 13719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e: Generated new TSK 0
I20260807 18:32:42.710918 13719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:32:42.792166 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34163
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:43.141006 13731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:43.141333 13731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:43.141501 13731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:43.149322 13731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:43.149470 13731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:43.149637 13731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:32:43.159626 13731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:43.161005 13731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:43.162400 13731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:43.176345 13740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:43.176380 13737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:43.176292 13738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:43.177716 13731 server_base.cc:1034] running on GCE node
I20260807 18:32:43.178364 13731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:43.179440 13731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:43.180667 13731 hybrid_clock.cc:648] HybridClock initialized: now 1786127563180613 us; error 44 us; skew 500 ppm
I20260807 18:32:43.181051 13731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:43 dist-test-slave-1xrh krb5kdc[13619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127563, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:43.184787 13731 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:32:43.186228 13731 webserver.cc:460] Webserver started at http://127.27.1.193:40205/ using document root <none> and password file <none>
I20260807 18:32:43.186883 13731 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:43.186988 13731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:43.187371 13731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:43.190281 13731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "05460d4f4a8b46f3b663f9a9e702f641"
format_stamp: "Formatted at 2026-08-07 18:32:43 on dist-test-slave-1xrh"
server_key: "faf3dcf2e21723af313ad7749e6fc067"
server_key_iv: "875240ece2a1d37968087df3821e7247"
server_key_version: "encryptionkey@0"
I20260807 18:32:43.191004 13731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "05460d4f4a8b46f3b663f9a9e702f641"
format_stamp: "Formatted at 2026-08-07 18:32:43 on dist-test-slave-1xrh"
server_key: "faf3dcf2e21723af313ad7749e6fc067"
server_key_iv: "875240ece2a1d37968087df3821e7247"
server_key_version: "encryptionkey@0"
I20260807 18:32:43.196203 13731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:32:43.199815 13748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:43.201138 13731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:32:43.201269 13731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "05460d4f4a8b46f3b663f9a9e702f641"
format_stamp: "Formatted at 2026-08-07 18:32:43 on dist-test-slave-1xrh"
server_key: "faf3dcf2e21723af313ad7749e6fc067"
server_key_iv: "875240ece2a1d37968087df3821e7247"
server_key_version: "encryptionkey@0"
I20260807 18:32:43.201428 13731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:43.215906 13731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:43.219559 13731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:43.219785 13731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:43.220647 13731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:43.221856 13731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:43.221925 13731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:43.222016 13731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:43.222065 13731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:43.255848 13731 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:40613
I20260807 18:32:43.255899 13864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:40613 every 8 connection(s)
I20260807 18:32:43.257189 13731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:32:43.265384 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 13731
I20260807 18:32:43.265645 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:32:43.266386 27655 external_mini_cluster.cc:1428] Setting key d0d9f6d8c83d09851b10fd5eb445ea4d
Aug 07 18:32:43 dist-test-slave-1xrh krb5kdc[13619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127563, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:43.279457 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:43.259834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:45135 (local address 127.27.1.254:34163)
0807 18:32:43.260301 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:43.260306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:43.261375 (+ 1069us) server_negotiation.cc:407] Connection header received
0807 18:32:43.263215 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:43.263220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:43.263315 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:43.263448 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:43.265080 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:43.266049 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:43.268396 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:43.268954 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:43.269108 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:43.272505 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:43.272514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:43.272529 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:43.272576 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:43.276155 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:43.276884 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:43.276892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:43.276898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:43.277058 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:43.277461 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:43.277468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:43.277474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:43.277866 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:43.278027 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:43.278694 (+ 667us) server_negotiation.cc:299] Negotiation successful
0807 18:32:43.278955 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
I20260807 18:32:43.280139 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:43.260379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34163 (local address 127.27.1.193:45135)
0807 18:32:43.261199 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:43.261248 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:43.262903 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:43.263725 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:43.263734 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:43.264276 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:43.264853 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:43.264871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:43.266236 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:43.266244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:43.268233 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:43.268246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:43.269177 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:43.269183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:43.269382 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:43.270078 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:43.270100 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:43.272277 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:43.276292 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:43.276299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:43.276311 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:43.276751 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:43.277171 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:43.277174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:43.277176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:43.277351 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:43.277991 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:43.277997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:43.278309 (+ 312us) client_negotiation.cc:769] Sending connection context
0807 18:32:43.278540 (+ 231us) client_negotiation.cc:240] Negotiation successful
0807 18:32:43.278909 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":329,"threads_started":1}
I20260807 18:32:43.281716 13865 heartbeater.cc:347] Connected to a master server at 127.27.1.254:34163
I20260807 18:32:43.282030 13865 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:43.282743 13865 heartbeater.cc:511] Master 127.27.1.254:34163 requested a full tablet report, sending...
I20260807 18:32:43.284868 13668 ts_manager.cc:194] Registered new tserver with Master: 05460d4f4a8b46f3b663f9a9e702f641 (127.27.1.193:40613)
I20260807 18:32:43.286285 13668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:45135
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:32:43.338799 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34163
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:43.685588 13871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:43.685864 13871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:43.685958 13871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:43.693274 13871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:43.693387 13871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:43.693499 13871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:32:43.703262 13871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:43.704522 13871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:43.705814 13871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:43.719318 13876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:43.719280 13877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:43.719403 13880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:43.720477 13871 server_base.cc:1034] running on GCE node
I20260807 18:32:43.721086 13871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:43.721992 13871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:43.723229 13871 hybrid_clock.cc:648] HybridClock initialized: now 1786127563723195 us; error 38 us; skew 500 ppm
I20260807 18:32:43.723564 13871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:43 dist-test-slave-1xrh krb5kdc[13619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127563, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:43.726745 13871 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:32:43.728260 13871 webserver.cc:460] Webserver started at http://127.27.1.194:42407/ using document root <none> and password file <none>
I20260807 18:32:43.728931 13871 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:43.729084 13871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:43.729416 13871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:43.732482 13871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "34cb134855494ad68b77792b1a17453c"
format_stamp: "Formatted at 2026-08-07 18:32:43 on dist-test-slave-1xrh"
server_key: "e8758a45615fb36067d845533f574aea"
server_key_iv: "f31ad278e4cf039ba3d33aa377569dbe"
server_key_version: "encryptionkey@0"
I20260807 18:32:43.733258 13871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "34cb134855494ad68b77792b1a17453c"
format_stamp: "Formatted at 2026-08-07 18:32:43 on dist-test-slave-1xrh"
server_key: "e8758a45615fb36067d845533f574aea"
server_key_iv: "f31ad278e4cf039ba3d33aa377569dbe"
server_key_version: "encryptionkey@0"
I20260807 18:32:43.738325 13871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 18:32:43.741801 13889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:43.743112 13871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 18:32:43.743340 13871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "34cb134855494ad68b77792b1a17453c"
format_stamp: "Formatted at 2026-08-07 18:32:43 on dist-test-slave-1xrh"
server_key: "e8758a45615fb36067d845533f574aea"
server_key_iv: "f31ad278e4cf039ba3d33aa377569dbe"
server_key_version: "encryptionkey@0"
I20260807 18:32:43.743552 13871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:43.767974 13871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:43.772185 13871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:43.772500 13871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:43.773512 13871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:43.774899 13871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:43.775012 13871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:43.775205 13871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:43.775292 13871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 18:32:43.812515 13871 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:42739
I20260807 18:32:43.812568 14009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:42739 every 8 connection(s)
I20260807 18:32:43.813966 13871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:32:43.821609 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 13871
I20260807 18:32:43.821818 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:32:43.822278 27655 external_mini_cluster.cc:1428] Setting key c25fa06f4b75994a4df26f79157d60c0
Aug 07 18:32:43 dist-test-slave-1xrh krb5kdc[13619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127563, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:43.835350 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:43.816730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:54569 (local address 127.27.1.254:34163)
0807 18:32:43.817290 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:43.817295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:43.818360 (+ 1065us) server_negotiation.cc:407] Connection header received
0807 18:32:43.820347 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:43.820353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:43.820456 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:43.820585 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:43.822247 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:43.823258 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:43.825001 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:43.825465 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:43.825606 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:43.829421 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:43.829426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:43.829430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:43.829462 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:43.832077 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:43.832835 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:43.832840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:43.832844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:43.833037 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:43.833489 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:43.833494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:43.833497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:43.833779 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:43.833967 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:43.834660 (+ 693us) server_negotiation.cc:299] Negotiation successful
0807 18:32:43.834914 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":314,"threads_started":1}
I20260807 18:32:43.836390 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:43.817375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34163 (local address 127.27.1.194:54569)
0807 18:32:43.818182 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:43.818221 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:43.820056 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:43.820830 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:43.820838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:43.821386 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:43.822068 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:43.822088 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:43.823427 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:43.823431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:43.824840 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:43.824853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:43.825717 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:43.825726 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:43.825965 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:43.826748 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:43.826772 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:43.829170 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:43.832233 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:43.832241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:43.832260 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:43.832705 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:43.833160 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:43.833166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:43.833169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:43.833363 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:43.833932 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:43.833941 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:43.834289 (+ 348us) client_negotiation.cc:769] Sending connection context
0807 18:32:43.834578 (+ 289us) client_negotiation.cc:240] Negotiation successful
0807 18:32:43.835193 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":310,"threads_started":1}
I20260807 18:32:43.837996 14010 heartbeater.cc:347] Connected to a master server at 127.27.1.254:34163
I20260807 18:32:43.838307 14010 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:43.839035 14010 heartbeater.cc:511] Master 127.27.1.254:34163 requested a full tablet report, sending...
I20260807 18:32:43.840615 13668 ts_manager.cc:194] Registered new tserver with Master: 34cb134855494ad68b77792b1a17453c (127.27.1.194:42739)
I20260807 18:32:43.841881 13668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:54569
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:32:43.892652 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34163
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:44.240473 14016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:44.240784 14016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:44.240898 14016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:44.249217 14016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:44.249336 14016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:44.249485 14016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:32:44.263477 14016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:44.264760 14016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:44.266284 14016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:44.280189 14022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:44.280393 14024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:44.280189 14021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:44.281431 14016 server_base.cc:1034] running on GCE node
I20260807 18:32:44.282052 14016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:44.282934 14016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:44.284202 14016 hybrid_clock.cc:648] HybridClock initialized: now 1786127564284154 us; error 58 us; skew 500 ppm
I20260807 18:32:44.284598 14016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:44 dist-test-slave-1xrh krb5kdc[13619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127564, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:44.288002 14016 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:32:44.289113 13865 heartbeater.cc:503] Master 127.27.1.254:34163 was elected leader, sending a full tablet report...
I20260807 18:32:44.289677 14016 webserver.cc:460] Webserver started at http://127.27.1.195:39221/ using document root <none> and password file <none>
I20260807 18:32:44.290462 14016 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:44.290572 14016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:44.291051 14016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:44.294132 14016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "ba50678d52674385a3e72583837419df"
format_stamp: "Formatted at 2026-08-07 18:32:44 on dist-test-slave-1xrh"
server_key: "7d04ee09c77d49582f065f2cefa85ea3"
server_key_iv: "1a55d656d882140b9ef79a8d6a430748"
server_key_version: "encryptionkey@0"
I20260807 18:32:44.294894 14016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "ba50678d52674385a3e72583837419df"
format_stamp: "Formatted at 2026-08-07 18:32:44 on dist-test-slave-1xrh"
server_key: "7d04ee09c77d49582f065f2cefa85ea3"
server_key_iv: "1a55d656d882140b9ef79a8d6a430748"
server_key_version: "encryptionkey@0"
I20260807 18:32:44.300635 14016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 18:32:44.304188 14032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:44.305630 14016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:32:44.305770 14016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "ba50678d52674385a3e72583837419df"
format_stamp: "Formatted at 2026-08-07 18:32:44 on dist-test-slave-1xrh"
server_key: "7d04ee09c77d49582f065f2cefa85ea3"
server_key_iv: "1a55d656d882140b9ef79a8d6a430748"
server_key_version: "encryptionkey@0"
I20260807 18:32:44.305953 14016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:44.333153 14016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:44.337548 14016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:44.337795 14016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:44.338699 14016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:44.340132 14016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:44.340204 14016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:44.340287 14016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:44.340333 14016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:44.374991 14016 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:44891
I20260807 18:32:44.375090 14150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:44891 every 8 connection(s)
I20260807 18:32:44.376335 14016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:32:44.382915 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 14016
I20260807 18:32:44.383101 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:32:44.383579 27655 external_mini_cluster.cc:1428] Setting key 572ec423ed576372052c7506c5827489
Aug 07 18:32:44 dist-test-slave-1xrh krb5kdc[13619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127564, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:44.397419 14153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.378990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:36691 (local address 127.27.1.254:34163)
0807 18:32:44.379531 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:44.379537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:44.380490 (+ 953us) server_negotiation.cc:407] Connection header received
0807 18:32:44.382322 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:44.382326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:44.382398 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:44.382501 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:44.384385 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.385605 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.387760 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.388156 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.388302 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.391797 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:44.391802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:44.391805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:44.391837 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:44.394298 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:44.395030 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:44.395037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:44.395041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:44.395238 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:44.395674 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:44.395679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:44.395683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:44.395944 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:44.396123 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:44.396792 (+ 669us) server_negotiation.cc:299] Negotiation successful
0807 18:32:44.397012 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":257,"threads_started":1}
I20260807 18:32:44.398559 14154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.379592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34163 (local address 127.27.1.195:36691)
0807 18:32:44.380307 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:44.380354 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:44.382047 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:44.382729 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:44.382739 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:44.383441 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:44.384155 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.384181 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.385775 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.385779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.387539 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.387552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.388403 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.388413 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.388682 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.389391 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:44.389413 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:44.391579 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:44.394447 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:44.394456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:44.394473 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:44.394894 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:44.395355 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:44.395360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:44.395364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:44.395546 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:44.396096 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:44.396104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:44.396420 (+ 316us) client_negotiation.cc:769] Sending connection context
0807 18:32:44.396685 (+ 265us) client_negotiation.cc:240] Negotiation successful
0807 18:32:44.397211 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":272,"threads_started":1}
I20260807 18:32:44.400064 14151 heartbeater.cc:347] Connected to a master server at 127.27.1.254:34163
I20260807 18:32:44.400377 14151 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:44.401085 14151 heartbeater.cc:511] Master 127.27.1.254:34163 requested a full tablet report, sending...
I20260807 18:32:44.402487 13668 ts_manager.cc:194] Registered new tserver with Master: ba50678d52674385a3e72583837419df (127.27.1.195:44891)
I20260807 18:32:44.403367 13668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:36691
I20260807 18:32:44.411332 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:32:44.428197 14153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.415935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.27.1.254:34163)
0807 18:32:44.416101 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:44.416106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:44.416621 (+ 515us) server_negotiation.cc:407] Connection header received
0807 18:32:44.416749 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:44.416752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:44.416814 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:44.416895 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:44.418339 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.419079 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.420530 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.420934 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.421027 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.423447 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:44.423452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:44.423455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:44.423485 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:44.425976 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:44.426557 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:44.426561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:44.426564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:44.426678 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:44.427003 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:44.427006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:44.427008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:44.427321 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:44.427503 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:44.427825 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 18:32:44.427985 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 18:32:44.431954 13668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:32:44.438674 13668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:32:44.461344 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.450688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:44891 (local address 127.0.0.1:39278)
0807 18:32:44.453323 (+ 2635us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:44.453341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:44.453463 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:44.454469 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:44.454474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:44.454498 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:44.454964 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.454977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.456611 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.456616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.458857 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.458870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.460247 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.460254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.460535 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.460590 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 18:32:44.460730 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 18:32:44.460927 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2498,"mutex_wait_us":26,"thread_start_us":1107,"threads_started":1}
I20260807 18:32:44.461364 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.451408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.27.1.195:44891)
0807 18:32:44.453049 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:44.453059 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:44.453621 (+ 562us) server_negotiation.cc:407] Connection header received
0807 18:32:44.453729 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:44.453736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:44.453980 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:44.454169 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:44.455220 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.456432 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.459002 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.460027 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.460188 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.460439 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:44.460810 (+ 371us) server_negotiation.cc:299] Negotiation successful
0807 18:32:44.460989 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":1270,"threads_started":1}
I20260807 18:32:44.462118 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.450402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40613 (local address 127.0.0.1:54668)
0807 18:32:44.451558 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:44.451608 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:44.451827 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:44.453346 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:44.453356 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:44.453394 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:44.453817 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.453830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.455831 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.455836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.458726 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.458743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.460922 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.460929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.461429 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.461469 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 18:32:44.461579 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 18:32:44.461706 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":71,"thread_start_us":2638,"threads_started":1}
I20260807 18:32:44.463094 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.450857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.27.1.193:40613)
0807 18:32:44.452640 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:44.452653 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:44.452688 (+ 35us) server_negotiation.cc:407] Connection header received
0807 18:32:44.452807 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:44.452817 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:44.453110 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:44.453313 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:44.454037 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.455684 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.458928 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.460388 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.460535 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.460771 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:44.462588 (+ 1817us) server_negotiation.cc:299] Negotiation successful
0807 18:32:44.462711 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1564,"thread_start_us":1383,"threads_started":1}
I20260807 18:32:44.464123 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.454189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:42739 (local address 127.0.0.1:44396)
0807 18:32:44.454780 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:44.454799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:44.454991 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:44.455832 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:44.455836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:44.455862 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:44.456355 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.456366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.457988 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.457994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.460315 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.460331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.463286 (+ 2955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.463292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.463466 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.463510 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 18:32:44.463610 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 18:32:44.463729 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":29,"thread_start_us":303,"threads_started":1}
I20260807 18:32:44.464202 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.453631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.27.1.194:42739)
0807 18:32:44.454185 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:44.454193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:44.454896 (+ 703us) server_negotiation.cc:407] Connection header received
0807 18:32:44.455210 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:44.455217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:44.455429 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:44.455583 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:44.456537 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.457828 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.460540 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.462015 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.462173 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.462365 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:44.463682 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0807 18:32:44.463873 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":284,"threads_started":1}
I20260807 18:32:44.465718 14085 tablet_service.cc:1467] Processing CreateTablet for tablet c360757ecc3a4c09af9fc4b9c338cb57 (DEFAULT_TABLE table=test-table [id=1ad5ff9e66e04faebb2bde5ca366b561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:44.467365 14085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c360757ecc3a4c09af9fc4b9c338cb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:44.468847 13798 tablet_service.cc:1467] Processing CreateTablet for tablet c360757ecc3a4c09af9fc4b9c338cb57 (DEFAULT_TABLE table=test-table [id=1ad5ff9e66e04faebb2bde5ca366b561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:44.470288 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c360757ecc3a4c09af9fc4b9c338cb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:44.470072 13942 tablet_service.cc:1467] Processing CreateTablet for tablet c360757ecc3a4c09af9fc4b9c338cb57 (DEFAULT_TABLE table=test-table [id=1ad5ff9e66e04faebb2bde5ca366b561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:44.471117 13942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c360757ecc3a4c09af9fc4b9c338cb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:44.481987 14170 tablet_bootstrap.cc:492] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: Bootstrap starting.
I20260807 18:32:44.485148 14172 tablet_bootstrap.cc:492] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: Bootstrap starting.
I20260807 18:32:44.486531 14170 tablet_bootstrap.cc:654] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:44.486941 14171 tablet_bootstrap.cc:492] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: Bootstrap starting.
I20260807 18:32:44.487931 14170 log.cc:824] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:44.488858 14172 tablet_bootstrap.cc:654] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:44.490017 14172 log.cc:824] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:44.490166 14171 tablet_bootstrap.cc:654] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:44.491204 14170 tablet_bootstrap.cc:492] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: No bootstrap required, opened a new log
I20260807 18:32:44.491421 14171 log.cc:824] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:44.491518 14170 ts_tablet_manager.cc:1397] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 18:32:44.493042 14172 tablet_bootstrap.cc:492] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: No bootstrap required, opened a new log
I20260807 18:32:44.493288 14172 ts_tablet_manager.cc:1397] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 18:32:44.494076 14171 tablet_bootstrap.cc:492] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: No bootstrap required, opened a new log
I20260807 18:32:44.494347 14171 ts_tablet_manager.cc:1397] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260807 18:32:44.501179 14171 raft_consensus.cc:348] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 } } peers { permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 } } peers { permanent_uuid: "34cb134855494ad68b77792b1a17453c" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 42739 } }
I20260807 18:32:44.501554 14171 raft_consensus.cc:374] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:44.501688 14171 raft_consensus.cc:729] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba50678d52674385a3e72583837419df, State: Initialized, Role: FOLLOWER
I20260807 18:32:44.502302 14171 consensus_queue.cc:260] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 } } peers { permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 } } peers { permanent_uuid: "34cb134855494ad68b77792b1a17453c" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 42739 } }
I20260807 18:32:44.502750 14170 raft_consensus.cc:348] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 } } peers { permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 } } peers { permanent_uuid: "34cb134855494ad68b77792b1a17453c" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 42739 } }
I20260807 18:32:44.503233 14170 raft_consensus.cc:374] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:44.503377 14170 raft_consensus.cc:729] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34cb134855494ad68b77792b1a17453c, State: Initialized, Role: FOLLOWER
I20260807 18:32:44.503448 14151 heartbeater.cc:503] Master 127.27.1.254:34163 was elected leader, sending a full tablet report...
I20260807 18:32:44.504031 14170 consensus_queue.cc:260] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 } } peers { permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 } } peers { permanent_uuid: "34cb134855494ad68b77792b1a17453c" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 42739 } }
I20260807 18:32:44.503927 14172 raft_consensus.cc:348] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 } } peers { permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 } } peers { permanent_uuid: "34cb134855494ad68b77792b1a17453c" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 42739 } }
I20260807 18:32:44.504237 14171 ts_tablet_manager.cc:1428] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260807 18:32:44.504333 14172 raft_consensus.cc:374] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:44.504431 14172 raft_consensus.cc:729] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05460d4f4a8b46f3b663f9a9e702f641, State: Initialized, Role: FOLLOWER
I20260807 18:32:44.505234 14172 consensus_queue.cc:260] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 } } peers { permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 } } peers { permanent_uuid: "34cb134855494ad68b77792b1a17453c" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 42739 } }
I20260807 18:32:44.507501 14172 ts_tablet_manager.cc:1428] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260807 18:32:44.508752 14010 heartbeater.cc:503] Master 127.27.1.254:34163 was elected leader, sending a full tablet report...
W20260807 18:32:44.509583 13866 tablet.cc:2369] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:32:44.510366 14170 ts_tablet_manager.cc:1428] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: Time spent starting tablet: real 0.019s user 0.009s sys 0.008s
W20260807 18:32:44.566334 14011 tablet.cc:2369] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:32:44.627998 14152 tablet.cc:2369] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:32:44.636691 14178 raft_consensus.cc:482] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:32:44.637030 14178 raft_consensus.cc:504] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 } } peers { permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 } } peers { permanent_uuid: "34cb134855494ad68b77792b1a17453c" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 42739 } }
I20260807 18:32:44.638754 14178 leader_election.cc:288] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05460d4f4a8b46f3b663f9a9e702f641 (127.27.1.193:40613), ba50678d52674385a3e72583837419df (127.27.1.195:44891)
I20260807 18:32:44.646409 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.639165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:36331 (local address 127.27.1.193:40613)
0807 18:32:44.639434 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:44.639444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:44.640278 (+ 834us) server_negotiation.cc:407] Connection header received
0807 18:32:44.640561 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:44.640569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:44.640650 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:44.640718 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:44.641373 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.642108 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.644305 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.645379 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.645571 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.645656 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:44.646015 (+ 359us) server_negotiation.cc:299] Negotiation successful
0807 18:32:44.646155 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 18:32:44.646450 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.639528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:44055 (local address 127.27.1.195:44891)
0807 18:32:44.639740 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:44.639748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:44.640424 (+ 676us) server_negotiation.cc:407] Connection header received
0807 18:32:44.640609 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:44.640612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:44.640671 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:44.640782 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:44.641453 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.642216 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.644118 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.645253 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.645434 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.645513 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:44.646081 (+ 568us) server_negotiation.cc:299] Negotiation successful
0807 18:32:44.646204 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 18:32:44.646595 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.639019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:40613 (local address 127.27.1.194:36331)
0807 18:32:44.640138 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:44.640158 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:44.640357 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:44.640770 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:44.640774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:44.640799 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:44.641220 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.641230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.642244 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.642252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.644136 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.644152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.645632 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.645642 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.645847 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.645876 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 18:32:44.645989 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 18:32:44.646112 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":96,"thread_start_us":984,"threads_started":1}
I20260807 18:32:44.646678 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.639015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:44891 (local address 127.27.1.194:44055)
0807 18:32:44.640321 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:44.640335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:44.640470 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:44.640828 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:44.640832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:44.640863 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:44.641259 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.641268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.642356 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.642363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.643948 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:44.643960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.645652 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.645656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:44.645902 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.645940 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 18:32:44.646044 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 18:32:44.646163 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":60,"thread_start_us":712,"threads_started":1}
I20260807 18:32:44.647871 13819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c360757ecc3a4c09af9fc4b9c338cb57" candidate_uuid: "34cb134855494ad68b77792b1a17453c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05460d4f4a8b46f3b663f9a9e702f641" is_pre_election: true
I20260807 18:32:44.647909 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c360757ecc3a4c09af9fc4b9c338cb57" candidate_uuid: "34cb134855494ad68b77792b1a17453c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba50678d52674385a3e72583837419df" is_pre_election: true
I20260807 18:32:44.648504 14105 raft_consensus.cc:2454] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34cb134855494ad68b77792b1a17453c in term 0.
I20260807 18:32:44.648499 13819 raft_consensus.cc:2454] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34cb134855494ad68b77792b1a17453c in term 0.
I20260807 18:32:44.649228 13891 leader_election.cc:302] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05460d4f4a8b46f3b663f9a9e702f641, 34cb134855494ad68b77792b1a17453c, ba50678d52674385a3e72583837419df; no voters:
I20260807 18:32:44.649652 14178 raft_consensus.cc:2781] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:32:44.649806 14178 raft_consensus.cc:482] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:32:44.649919 14178 raft_consensus.cc:3037] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:44.652325 14178 raft_consensus.cc:504] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 } } peers { permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 } } peers { permanent_uuid: "34cb134855494ad68b77792b1a17453c" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 42739 } }
I20260807 18:32:44.653066 14178 leader_election.cc:288] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [CANDIDATE]: Term 1 election: Requested vote from peers 05460d4f4a8b46f3b663f9a9e702f641 (127.27.1.193:40613), ba50678d52674385a3e72583837419df (127.27.1.195:44891)
I20260807 18:32:44.653478 13819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c360757ecc3a4c09af9fc4b9c338cb57" candidate_uuid: "34cb134855494ad68b77792b1a17453c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05460d4f4a8b46f3b663f9a9e702f641"
I20260807 18:32:44.653560 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c360757ecc3a4c09af9fc4b9c338cb57" candidate_uuid: "34cb134855494ad68b77792b1a17453c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba50678d52674385a3e72583837419df"
I20260807 18:32:44.653687 13819 raft_consensus.cc:3037] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:44.653759 14105 raft_consensus.cc:3037] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:44.655989 13819 raft_consensus.cc:2454] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34cb134855494ad68b77792b1a17453c in term 1.
I20260807 18:32:44.656002 14105 raft_consensus.cc:2454] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34cb134855494ad68b77792b1a17453c in term 1.
I20260807 18:32:44.656417 13890 leader_election.cc:302] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05460d4f4a8b46f3b663f9a9e702f641, 34cb134855494ad68b77792b1a17453c, ba50678d52674385a3e72583837419df; no voters:
I20260807 18:32:44.656764 14178 raft_consensus.cc:2781] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:32:44.657225 14178 raft_consensus.cc:686] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 1 LEADER]: Becoming Leader. State: Replica: 34cb134855494ad68b77792b1a17453c, State: Running, Role: LEADER
I20260807 18:32:44.657694 14178 consensus_queue.cc:237] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 } } peers { permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 } } peers { permanent_uuid: "34cb134855494ad68b77792b1a17453c" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 42739 } }
I20260807 18:32:44.662577 13668 catalog_manager.cc:5495] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c reported cstate change: term changed from 0 to 1, leader changed from <none> to 34cb134855494ad68b77792b1a17453c (127.27.1.194). New cstate: current_term: 1 leader_uuid: "34cb134855494ad68b77792b1a17453c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34cb134855494ad68b77792b1a17453c" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 42739 } health_report { overall_health: HEALTHY } } }
I20260807 18:32:44.703895 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.697982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.27.1.194:42739)
0807 18:32:44.698214 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:44.698220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:44.698241 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:32:44.698422 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:44.698426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:44.698514 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:44.698678 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:32:44.699101 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.699903 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.701583 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.702030 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.702176 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.702687 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:32:44.703274 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:32:44.703449 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:44.703517 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 18:32:44.703600 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:32:44.712241 13819 raft_consensus.cc:1261] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641 [term 1 FOLLOWER]: Refusing update from remote peer 34cb134855494ad68b77792b1a17453c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:32:44.712366 14105 raft_consensus.cc:1261] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df [term 1 FOLLOWER]: Refusing update from remote peer 34cb134855494ad68b77792b1a17453c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:32:44.713312 14182 consensus_queue.cc:1035] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05460d4f4a8b46f3b663f9a9e702f641" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:32:44.713655 14178 consensus_queue.cc:1035] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba50678d52674385a3e72583837419df" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:32:44.724820 14185 mvcc.cc:205] Tried to move back new op lower bound from 7315978505051852800 to 7315978504841715712. Current Snapshot: MvccSnapshot[applied={T|T < 7315978505051852800}]
I20260807 18:32:44.732663 14186 mvcc.cc:205] Tried to move back new op lower bound from 7315978505051852800 to 7315978504841715712. Current Snapshot: MvccSnapshot[applied={T|T < 7315978505051852800}]
I20260807 18:32:44.739751 13668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:32:44.740036 13668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:32:44.743711 13668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62e2d60374554e94b14978978c34529e: Sending DeleteTablet for 3 replicas of tablet c360757ecc3a4c09af9fc4b9c338cb57
I20260807 18:32:44.744971 14085 tablet_service.cc:1514] Processing DeleteTablet for tablet c360757ecc3a4c09af9fc4b9c338cb57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:44 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:39278
I20260807 18:32:44.745036 13798 tablet_service.cc:1514] Processing DeleteTablet for tablet c360757ecc3a4c09af9fc4b9c338cb57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:44 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:54668
I20260807 18:32:44.745278 13942 tablet_service.cc:1514] Processing DeleteTablet for tablet c360757ecc3a4c09af9fc4b9c338cb57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:44 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:44396
I20260807 18:32:44.747597 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 13731
I20260807 18:32:44.762823 14196 tablet_replica.cc:330] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: stopping tablet replica
I20260807 18:32:44.763579 14196 raft_consensus.cc:2229] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:32:44.764405 14196 raft_consensus.cc:2258] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:44.764694 14198 tablet_replica.cc:330] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: stopping tablet replica
I20260807 18:32:44.764864 14197 tablet_replica.cc:330] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: stopping tablet replica
I20260807 18:32:44.765277 14197 raft_consensus.cc:2229] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:32:44.765748 14197 raft_consensus.cc:2258] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:44.767411 14196 ts_tablet_manager.cc:1905] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:44.767858 14198 raft_consensus.cc:2229] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:32:44.767908 14197 ts_tablet_manager.cc:1905] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:44.768589 14198 raft_consensus.cc:2258] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 07 18:32:44 dist-test-slave-1xrh krb5kdc[13619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:32:44.772989 14198 ts_tablet_manager.cc:1905] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:44.775578 14197 ts_tablet_manager.cc:1918] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:44.775815 14197 log.cc:1196] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/c360757ecc3a4c09af9fc4b9c338cb57
I20260807 18:32:44.776451 14197 ts_tablet_manager.cc:1939] T c360757ecc3a4c09af9fc4b9c338cb57 P 05460d4f4a8b46f3b663f9a9e702f641: Deleting consensus metadata
I20260807 18:32:44.778707 14196 ts_tablet_manager.cc:1918] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:44.778892 14196 log.cc:1196] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/c360757ecc3a4c09af9fc4b9c338cb57
I20260807 18:32:44.778824 13651 catalog_manager.cc:4841] TS 05460d4f4a8b46f3b663f9a9e702f641 (127.27.1.193:40613): tablet c360757ecc3a4c09af9fc4b9c338cb57 (table test-table [id=1ad5ff9e66e04faebb2bde5ca366b561]) successfully deleted
I20260807 18:32:44.779481 14196 ts_tablet_manager.cc:1939] T c360757ecc3a4c09af9fc4b9c338cb57 P ba50678d52674385a3e72583837419df: Deleting consensus metadata
I20260807 18:32:44.781247 13652 catalog_manager.cc:4841] TS ba50678d52674385a3e72583837419df (127.27.1.195:44891): tablet c360757ecc3a4c09af9fc4b9c338cb57 (table test-table [id=1ad5ff9e66e04faebb2bde5ca366b561]) successfully deleted
I20260807 18:32:44.782332 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.749254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.27.1.193:40613)
0807 18:32:44.749455 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:44.749464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:44.754050 (+ 4586us) server_negotiation.cc:407] Connection header received
0807 18:32:44.754999 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:44.755008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:44.755143 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:44.755237 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:44.759711 (+ 4474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.761274 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.765638 (+ 4364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.766453 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.766669 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.773569 (+ 6900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:44.773585 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:44.773607 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:44.773658 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:44.777712 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:44.779500 (+ 1788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:44.779510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:44.779514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:44.779752 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:44.780255 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:44.780260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:44.780264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:44.780728 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:44.781027 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:44.781686 (+ 659us) server_negotiation.cc:299] Negotiation successful
0807 18:32:44.781929 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:32:44.783922 13860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:44.786176 14198 ts_tablet_manager.cc:1918] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:44.786429 14198 log.cc:1196] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/c360757ecc3a4c09af9fc4b9c338cb57
I20260807 18:32:44.787369 14198 ts_tablet_manager.cc:1939] T c360757ecc3a4c09af9fc4b9c338cb57 P 34cb134855494ad68b77792b1a17453c: Deleting consensus metadata
I20260807 18:32:44.789443 13651 catalog_manager.cc:4841] TS 34cb134855494ad68b77792b1a17453c (127.27.1.194:42739): tablet c360757ecc3a4c09af9fc4b9c338cb57 (table test-table [id=1ad5ff9e66e04faebb2bde5ca366b561]) successfully deleted
I20260807 18:32:44.909456 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 13731
I20260807 18:32:44.930505 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 13871
Aug 07 18:32:44 dist-test-slave-1xrh krb5kdc[13619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:32:44.947103 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:44.931365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.27.1.194:42739)
0807 18:32:44.931545 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:44.931551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:44.931593 (+ 42us) server_negotiation.cc:407] Connection header received
0807 18:32:44.931813 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:44.931819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:44.931902 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:44.932028 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:44.933342 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.934359 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.935854 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:44.936316 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:44.936464 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:44.941042 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:44.941058 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:44.941074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:44.941118 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:44.944202 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:44.944939 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:44.944950 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:44.944952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:44.945099 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:44.945499 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:44.945504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:44.945512 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:44.945849 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:44.946212 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:44.946516 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 18:32:44.946738 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 18:32:44.947871 14005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:45.063956 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 13871
I20260807 18:32:45.085619 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 14016
Aug 07 18:32:45 dist-test-slave-1xrh krb5kdc[13619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:32:45.100513 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:45.086362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.27.1.195:44891)
0807 18:32:45.086535 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:45.086539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:45.086557 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:32:45.086710 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:45.086713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:45.086781 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:45.086878 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:45.088229 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:45.088943 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:45.090144 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:45.090517 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:45.090608 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:45.095005 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:45.095026 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:45.095044 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:45.095079 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:45.097846 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:45.098550 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:45.098557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:45.098559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:45.098716 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:45.099039 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:45.099042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:45.099044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:45.099482 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:45.099706 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:45.099965 (+ 259us) server_negotiation.cc:299] Negotiation successful
0807 18:32:45.100125 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 18:32:45.101208 14146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:45.191521 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 14016
I20260807 18:32:45.211338 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 13633
I20260807 18:32:45.212517 13698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:45.325188 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 13633
2026-08-07T18:32:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4732 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T18:32:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:32:45Z Disabled control of system clock
I20260807 18:32:45.424774 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:46479
--webserver_interface=127.27.1.254
--webserver_port=0
--builtin_ntp_servers=127.27.1.212:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 18:32:45.791625 14209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:45.791977 14209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:45.792151 14209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:45.799494 14209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:32:45.799638 14209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:45.799734 14209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:45.799830 14209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:32:45.799924 14209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:32:45.809418 14209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:46479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:45.811187 14209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:45.814596 14209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:45.828364 14215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:45.828475 14214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:45.828593 14218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:45.829663 14209 server_base.cc:1034] running on GCE node
I20260807 18:32:45.830511 14209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:45.832108 14209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:45.833361 14209 hybrid_clock.cc:648] HybridClock initialized: now 1786127565833305 us; error 41 us; skew 500 ppm
I20260807 18:32:45.833740 14209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:32:45.841892 14209 webserver.cc:460] Webserver started at http://127.27.1.254:42451/ using document root <none> and password file <none>
I20260807 18:32:45.842684 14209 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:45.842844 14209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:45.843304 14209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:45.846596 14209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "6eebd303cedf4248b70aee1fcf3fd12d"
format_stamp: "Formatted at 2026-08-07 18:32:45 on dist-test-slave-1xrh"
server_key: "75f5aedef2e5b3ff4c63653b546d1677"
server_key_iv: "8eb99df60961896b697ffc21ef46aca1"
server_key_version: "encryptionkey@0"
I20260807 18:32:45.847582 14209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "6eebd303cedf4248b70aee1fcf3fd12d"
format_stamp: "Formatted at 2026-08-07 18:32:45 on dist-test-slave-1xrh"
server_key: "75f5aedef2e5b3ff4c63653b546d1677"
server_key_iv: "8eb99df60961896b697ffc21ef46aca1"
server_key_version: "encryptionkey@0"
I20260807 18:32:45.853124 14209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:32:45.857013 14224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:45.858371 14209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:32:45.858562 14209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "6eebd303cedf4248b70aee1fcf3fd12d"
format_stamp: "Formatted at 2026-08-07 18:32:45 on dist-test-slave-1xrh"
server_key: "75f5aedef2e5b3ff4c63653b546d1677"
server_key_iv: "8eb99df60961896b697ffc21ef46aca1"
server_key_version: "encryptionkey@0"
I20260807 18:32:45.858783 14209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:45.909927 14209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:45.910878 14209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:45.911216 14209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:45.929796 14209 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:46479
I20260807 18:32:45.929818 14276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:46479 every 8 connection(s)
I20260807 18:32:45.931221 14209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:32:45.934695 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 14209
I20260807 18:32:45.934906 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:32:45.935182 14277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:45.935503 27655 external_mini_cluster.cc:1428] Setting key 5fdf84f4d8cf99d566494f117e473c5d
I20260807 18:32:45.950212 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:45.937483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35354 (local address 127.27.1.254:46479)
0807 18:32:45.938548 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:45.938575 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:45.938620 (+ 45us) server_negotiation.cc:407] Connection header received
0807 18:32:45.940269 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:45.940290 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:45.940738 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:45.941142 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:45.941734 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:45.943310 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:45.945507 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:45.946238 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:45.946382 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:45.947862 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:45.947870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:45.947887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:45.948455 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:45.948546 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:45.948570 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:45.948657 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:45.948914 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 18:32:45.949244 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":408,"threads_started":1}
I20260807 18:32:45.950551 14277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d: Bootstrap starting.
I20260807 18:32:45.954346 14277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:45.955374 14277 log.cc:824] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:45.958266 14277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d: No bootstrap required, opened a new log
I20260807 18:32:45.966012 14277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eebd303cedf4248b70aee1fcf3fd12d" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46479 } }
I20260807 18:32:45.966367 14277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:45.966455 14277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eebd303cedf4248b70aee1fcf3fd12d, State: Initialized, Role: FOLLOWER
I20260807 18:32:45.966939 14277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eebd303cedf4248b70aee1fcf3fd12d" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46479 } }
I20260807 18:32:45.967206 14277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:32:45.967324 14277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:32:45.967449 14277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:45.969549 14277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eebd303cedf4248b70aee1fcf3fd12d" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46479 } }
I20260807 18:32:45.970005 14277 leader_election.cc:302] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eebd303cedf4248b70aee1fcf3fd12d; no voters:
I20260807 18:32:45.970618 14277 leader_election.cc:288] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:32:45.970772 14282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:32:45.971935 14282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [term 1 LEADER]: Becoming Leader. State: Replica: 6eebd303cedf4248b70aee1fcf3fd12d, State: Running, Role: LEADER
I20260807 18:32:45.972524 14282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eebd303cedf4248b70aee1fcf3fd12d" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46479 } }
I20260807 18:32:45.972929 14277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:32:45.975570 14283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eebd303cedf4248b70aee1fcf3fd12d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eebd303cedf4248b70aee1fcf3fd12d" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46479 } } }
I20260807 18:32:45.976084 14283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:45.976686 14284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eebd303cedf4248b70aee1fcf3fd12d. Latest consensus state: current_term: 1 leader_uuid: "6eebd303cedf4248b70aee1fcf3fd12d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eebd303cedf4248b70aee1fcf3fd12d" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46479 } } }
I20260807 18:32:45.977051 14284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:45.978578 14291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:32:45.984727 14291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:32:45.993726 14291 catalog_manager.cc:1347] Generated new cluster ID: 5f321d7ea96548c0a64e2801f857ff8e
I20260807 18:32:45.993963 14291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:32:46.006242 14291 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:32:46.007789 14291 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:32:46.015244 14291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d: Generated new TSK 0
I20260807 18:32:46.016198 14291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:32:46.027935 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46479
--builtin_ntp_servers=127.27.1.212:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 18:32:46.371855 14301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:46.372128 14301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:46.372294 14301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:46.380255 14301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:46.380424 14301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:46.380609 14301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:32:46.389856 14301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:46.391183 14301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:46.393016 14301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:46.406410 14309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:46.406519 14307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:46.406481 14306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:46.407095 14301 server_base.cc:1034] running on GCE node
I20260807 18:32:46.407727 14301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:46.408685 14301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:46.409914 14301 hybrid_clock.cc:648] HybridClock initialized: now 1786127566409855 us; error 58 us; skew 500 ppm
I20260807 18:32:46.410226 14301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:32:46.412510 14301 webserver.cc:460] Webserver started at http://127.27.1.193:46729/ using document root <none> and password file <none>
I20260807 18:32:46.413185 14301 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:46.413314 14301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:46.413650 14301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:46.416447 14301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "53d9a71fd2b04e60b630057d92bfaebd"
format_stamp: "Formatted at 2026-08-07 18:32:46 on dist-test-slave-1xrh"
server_key: "bfd2192fae56ba231d0a44d8d322e65b"
server_key_iv: "150146db0ba5913383cd3ece85cf1cf0"
server_key_version: "encryptionkey@0"
I20260807 18:32:46.417236 14301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "53d9a71fd2b04e60b630057d92bfaebd"
format_stamp: "Formatted at 2026-08-07 18:32:46 on dist-test-slave-1xrh"
server_key: "bfd2192fae56ba231d0a44d8d322e65b"
server_key_iv: "150146db0ba5913383cd3ece85cf1cf0"
server_key_version: "encryptionkey@0"
I20260807 18:32:46.422401 14301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 18:32:46.425784 14316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:46.427055 14301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:32:46.427269 14301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "53d9a71fd2b04e60b630057d92bfaebd"
format_stamp: "Formatted at 2026-08-07 18:32:46 on dist-test-slave-1xrh"
server_key: "bfd2192fae56ba231d0a44d8d322e65b"
server_key_iv: "150146db0ba5913383cd3ece85cf1cf0"
server_key_version: "encryptionkey@0"
I20260807 18:32:46.427484 14301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:46.444099 14301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:46.444887 14301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:46.445201 14301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:46.446128 14301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:46.447665 14301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:46.447793 14301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:46.447923 14301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:46.448021 14301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:46.481680 14301 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:44161
I20260807 18:32:46.481730 14433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:44161 every 8 connection(s)
I20260807 18:32:46.483021 14301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:32:46.488227 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 14301
I20260807 18:32:46.488411 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:32:46.488847 27655 external_mini_cluster.cc:1428] Setting key 95f83305847c900937206ef2f908cc71
I20260807 18:32:46.491875 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46479
--builtin_ntp_servers=127.27.1.212:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 18:32:46.496263 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:46.485729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:40491 (local address 127.27.1.254:46479)
0807 18:32:46.486175 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:46.486181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:46.487483 (+ 1302us) server_negotiation.cc:407] Connection header received
0807 18:32:46.489492 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:46.489497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:46.489600 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:46.489735 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:46.491136 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:46.491886 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:46.493465 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:46.493872 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:46.494006 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:46.495423 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:46.495427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:46.495431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:46.495449 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:46.495494 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:46.495506 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:46.495605 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:46.495833 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 18:32:46.495929 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":205,"threads_started":1}
I20260807 18:32:46.497237 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:46.486380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:46479 (local address 127.27.1.193:40491)
0807 18:32:46.487330 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:46.487376 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:46.489197 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:46.489980 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:46.490000 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:46.490110 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:46.490923 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:46.490951 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:46.492049 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:46.492055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:46.493306 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:46.493318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:46.494114 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:46.494121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:46.494341 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:46.495052 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:32:46.495055 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:46.495093 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:32:46.495229 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:46.495605 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:46.495619 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:46.495628 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 18:32:46.495818 (+ 190us) client_negotiation.cc:240] Negotiation successful
0807 18:32:46.496211 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":290,"threads_started":1}
I20260807 18:32:46.498649 14434 heartbeater.cc:347] Connected to a master server at 127.27.1.254:46479
I20260807 18:32:46.498950 14434 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:46.499742 14434 heartbeater.cc:511] Master 127.27.1.254:46479 requested a full tablet report, sending...
I20260807 18:32:46.501775 14241 ts_manager.cc:194] Registered new tserver with Master: 53d9a71fd2b04e60b630057d92bfaebd (127.27.1.193:44161)
I20260807 18:32:46.503376 14241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.193:40491
W20260807 18:32:46.833197 14438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:46.833458 14438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:46.833550 14438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:46.840780 14438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:46.840873 14438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:46.840993 14438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:32:46.849859 14438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:46.851105 14438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:46.853021 14438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:46.866354 14443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:46.866333 14444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:46.866901 14446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:46.867203 14438 server_base.cc:1034] running on GCE node
I20260807 18:32:46.867805 14438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:46.868827 14438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:46.870074 14438 hybrid_clock.cc:648] HybridClock initialized: now 1786127566870019 us; error 61 us; skew 500 ppm
I20260807 18:32:46.870411 14438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:32:46.872879 14438 webserver.cc:460] Webserver started at http://127.27.1.194:36095/ using document root <none> and password file <none>
I20260807 18:32:46.873675 14438 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:46.873826 14438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:46.874125 14438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:46.877164 14438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "1b8a51f1a0194690be7acb3579c3e303"
format_stamp: "Formatted at 2026-08-07 18:32:46 on dist-test-slave-1xrh"
server_key: "129869377739a80d0cc2946031c5fb81"
server_key_iv: "f22dfe4bbb0c9bff26949a479651eb4f"
server_key_version: "encryptionkey@0"
I20260807 18:32:46.878006 14438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "1b8a51f1a0194690be7acb3579c3e303"
format_stamp: "Formatted at 2026-08-07 18:32:46 on dist-test-slave-1xrh"
server_key: "129869377739a80d0cc2946031c5fb81"
server_key_iv: "f22dfe4bbb0c9bff26949a479651eb4f"
server_key_version: "encryptionkey@0"
I20260807 18:32:46.883668 14438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 18:32:46.887308 14452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:46.888640 14438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:32:46.888810 14438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "1b8a51f1a0194690be7acb3579c3e303"
format_stamp: "Formatted at 2026-08-07 18:32:46 on dist-test-slave-1xrh"
server_key: "129869377739a80d0cc2946031c5fb81"
server_key_iv: "f22dfe4bbb0c9bff26949a479651eb4f"
server_key_version: "encryptionkey@0"
I20260807 18:32:46.889045 14438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:46.925163 14438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:46.926055 14438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:46.926390 14438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:46.927469 14438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:46.928926 14438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:46.929076 14438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:46.929198 14438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:46.929277 14438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:46.964607 14438 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:39977
I20260807 18:32:46.964666 14573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:39977 every 8 connection(s)
I20260807 18:32:46.965977 14438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:32:46.974094 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 14438
I20260807 18:32:46.974316 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:32:46.974905 27655 external_mini_cluster.cc:1428] Setting key 38b2431d5d13822726e8be4a1befd1ab
I20260807 18:32:46.977913 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46479
--builtin_ntp_servers=127.27.1.212:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 18:32:46.979195 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:46.968530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:47483 (local address 127.27.1.254:46479)
0807 18:32:46.968775 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:46.968782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:46.970130 (+ 1348us) server_negotiation.cc:407] Connection header received
0807 18:32:46.971840 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:46.971844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:46.971917 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:46.972011 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:46.973049 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:46.973754 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:46.976005 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:46.976464 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:46.976632 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:46.978208 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:46.978217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:46.978224 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:46.978249 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:46.978293 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:46.978312 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:46.978406 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:46.978641 (+ 235us) server_negotiation.cc:299] Negotiation successful
0807 18:32:46.978754 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 18:32:46.980319 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:46.969092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:46479 (local address 127.27.1.194:47483)
0807 18:32:46.969969 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:46.970015 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:46.971605 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:46.972213 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:46.972220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:46.972300 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:46.972899 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:46.972917 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:46.973914 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:46.973919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:46.975853 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:46.975871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:46.976746 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:46.976756 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:46.977004 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:46.977775 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:32:46.977778 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:46.977821 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:32:46.977909 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:46.978420 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:46.978430 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:46.978437 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 18:32:46.978596 (+ 159us) client_negotiation.cc:240] Negotiation successful
0807 18:32:46.979012 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
I20260807 18:32:46.981761 14574 heartbeater.cc:347] Connected to a master server at 127.27.1.254:46479
I20260807 18:32:46.982107 14574 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:46.983034 14574 heartbeater.cc:511] Master 127.27.1.254:46479 requested a full tablet report, sending...
I20260807 18:32:46.984678 14241 ts_manager.cc:194] Registered new tserver with Master: 1b8a51f1a0194690be7acb3579c3e303 (127.27.1.194:39977)
I20260807 18:32:46.985602 14241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.194:47483
W20260807 18:32:47.326668 14577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:47.327169 14577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:47.327292 14577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:47.334771 14577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:47.334870 14577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:47.334990 14577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:32:47.344543 14577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:46479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:47.345919 14577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:47.347779 14577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:47.361377 14582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:47.361472 14586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:47.361444 14583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:47.361974 14577 server_base.cc:1034] running on GCE node
I20260807 18:32:47.362690 14577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:47.363739 14577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:47.364995 14577 hybrid_clock.cc:648] HybridClock initialized: now 1786127567364911 us; error 94 us; skew 500 ppm
I20260807 18:32:47.365316 14577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:32:47.367751 14577 webserver.cc:460] Webserver started at http://127.27.1.195:41661/ using document root <none> and password file <none>
I20260807 18:32:47.368462 14577 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:47.368614 14577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:47.368940 14577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:47.371999 14577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "c00e7c1918fa4e67948670ccd382b006"
format_stamp: "Formatted at 2026-08-07 18:32:47 on dist-test-slave-1xrh"
server_key: "6c30dda12d7caea665691aa3b3888166"
server_key_iv: "1c2a04b75dc455785283e50cea79f764"
server_key_version: "encryptionkey@0"
I20260807 18:32:47.372922 14577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "c00e7c1918fa4e67948670ccd382b006"
format_stamp: "Formatted at 2026-08-07 18:32:47 on dist-test-slave-1xrh"
server_key: "6c30dda12d7caea665691aa3b3888166"
server_key_iv: "1c2a04b75dc455785283e50cea79f764"
server_key_version: "encryptionkey@0"
I20260807 18:32:47.378620 14577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 18:32:47.382411 14593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:47.383838 14577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 18:32:47.384033 14577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "c00e7c1918fa4e67948670ccd382b006"
format_stamp: "Formatted at 2026-08-07 18:32:47 on dist-test-slave-1xrh"
server_key: "6c30dda12d7caea665691aa3b3888166"
server_key_iv: "1c2a04b75dc455785283e50cea79f764"
server_key_version: "encryptionkey@0"
I20260807 18:32:47.384260 14577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:47.396030 14577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:47.396888 14577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:47.397202 14577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:47.398192 14577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:47.399716 14577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:47.399842 14577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:47.399974 14577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:47.400063 14577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:47.434233 14577 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:37879
I20260807 18:32:47.434412 14709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:37879 every 8 connection(s)
I20260807 18:32:47.435653 14577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:32:47.440527 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 14577
I20260807 18:32:47.440697 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:32:47.441164 27655 external_mini_cluster.cc:1428] Setting key 461af78b0756848c4f43308999a2ab4c
I20260807 18:32:47.449127 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.438303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:55793 (local address 127.27.1.254:46479)
0807 18:32:47.438496 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:47.438501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:47.439898 (+ 1397us) server_negotiation.cc:407] Connection header received
0807 18:32:47.442165 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:47.442169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:47.442237 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:47.442401 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:47.443890 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.444829 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.446408 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.446913 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.447022 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.448429 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:47.448433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:47.448437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:47.448455 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:47.448488 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:47.448500 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:47.448577 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:47.448785 (+ 208us) server_negotiation.cc:299] Negotiation successful
0807 18:32:47.448868 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 18:32:47.450157 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.438806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:46479 (local address 127.27.1.195:55793)
0807 18:32:47.439729 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:47.439765 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:47.441595 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:47.442703 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:47.442711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:47.442822 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:47.443694 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.443719 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.444970 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.444974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.446186 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.446199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.447115 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.447171 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.447389 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.448101 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:32:47.448103 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:47.448142 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:32:47.448220 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:47.448591 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:47.448599 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:47.448605 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 18:32:47.448740 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 18:32:47.449065 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":329,"threads_started":1}
I20260807 18:32:47.451669 14710 heartbeater.cc:347] Connected to a master server at 127.27.1.254:46479
I20260807 18:32:47.451936 14710 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:47.452600 14710 heartbeater.cc:511] Master 127.27.1.254:46479 requested a full tablet report, sending...
I20260807 18:32:47.453912 14241 ts_manager.cc:194] Registered new tserver with Master: c00e7c1918fa4e67948670ccd382b006 (127.27.1.195:37879)
I20260807 18:32:47.454754 14241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.195:55793
I20260807 18:32:47.456740 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:32:47.466919 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.460798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.27.1.254:46479)
0807 18:32:47.461018 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:47.461026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:47.461477 (+ 451us) server_negotiation.cc:407] Connection header received
0807 18:32:47.461644 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:47.461648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:47.461720 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:47.461847 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:47.462371 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.463067 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.464542 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.464942 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.465085 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.466205 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:47.466210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:47.466214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:47.466231 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:47.466263 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:47.466275 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:47.466373 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:47.466520 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 18:32:47.466610 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:32:47.470726 14240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:32:47.477855 14240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:32:47.506754 14434 heartbeater.cc:503] Master 127.27.1.254:46479 was elected leader, sending a full tablet report...
I20260807 18:32:47.515142 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.490971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:39977 (local address 127.0.0.1:56588)
0807 18:32:47.505126 (+ 14155us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:47.505143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:47.505278 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:47.506057 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:47.506064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:47.506102 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:47.506437 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.506451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.508403 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.508411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.511513 (+ 3102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.511529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.514223 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.514230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.514431 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.514485 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 18:32:47.514615 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 18:32:47.514784 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14027,"thread_start_us":11399,"threads_started":1}
I20260807 18:32:47.515477 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.491450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.27.1.194:39977)
0807 18:32:47.504867 (+ 13417us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:47.504877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:47.505452 (+ 575us) server_negotiation.cc:407] Connection header received
0807 18:32:47.505576 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:47.505582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:47.505836 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:47.506044 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:47.506635 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.508227 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.512242 (+ 4015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.513672 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.513821 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.514096 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:47.514701 (+ 605us) server_negotiation.cc:299] Negotiation successful
0807 18:32:47.514889 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13186,"thread_start_us":13025,"threads_started":1}
I20260807 18:32:47.516060 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.490669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:37879 (local address 127.0.0.1:59308)
0807 18:32:47.503543 (+ 12874us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:47.503601 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:47.503821 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:47.507919 (+ 4098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:47.507928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:47.507967 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:47.508560 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.508582 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.511306 (+ 2724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.511311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.513567 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.513583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.515308 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.515313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.515516 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.515558 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 18:32:47.515661 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 18:32:47.515797 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12546,"thread_start_us":14184,"threads_started":1}
I20260807 18:32:47.516940 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.490974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.27.1.195:37879)
0807 18:32:47.507357 (+ 16383us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:47.507365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:47.507395 (+ 30us) server_negotiation.cc:407] Connection header received
0807 18:32:47.507499 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:47.507506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:47.507711 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:47.507882 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:47.508884 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.511056 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.513781 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.514740 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.514869 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.515177 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:47.516441 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0807 18:32:47.516566 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16212,"thread_start_us":15426,"threads_started":1}
I20260807 18:32:47.519731 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.508202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44161 (local address 127.0.0.1:37604)
0807 18:32:47.511856 (+ 3654us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:47.511875 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:47.512032 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:47.513244 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:47.513248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:47.513274 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:47.513675 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.513687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.515567 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.515573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.517389 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.517402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.519087 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.519094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.519310 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.519349 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 18:32:47.519441 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 18:32:47.519537 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3512,"thread_start_us":3337,"threads_started":1}
I20260807 18:32:47.520450 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.505806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.27.1.193:44161)
0807 18:32:47.511528 (+ 5722us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:47.511536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:47.512725 (+ 1189us) server_negotiation.cc:407] Connection header received
0807 18:32:47.512822 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:47.512828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:47.513042 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:47.513227 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:47.513861 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.515385 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.517640 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.518884 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.519028 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.519315 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:47.519981 (+ 666us) server_negotiation.cc:299] Negotiation successful
0807 18:32:47.520171 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5530,"thread_start_us":5385,"threads_started":1}
I20260807 18:32:47.519910 14504 tablet_service.cc:1467] Processing CreateTablet for tablet 3e4921360ffc441593616b1f177aae21 (DEFAULT_TABLE table=test-table [id=41a6677b9c8a4fa3b9e93313ff207153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:47.520548 14642 tablet_service.cc:1467] Processing CreateTablet for tablet 3e4921360ffc441593616b1f177aae21 (DEFAULT_TABLE table=test-table [id=41a6677b9c8a4fa3b9e93313ff207153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:47.521504 14504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e4921360ffc441593616b1f177aae21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:47.521981 14642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e4921360ffc441593616b1f177aae21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:47.523434 14365 tablet_service.cc:1467] Processing CreateTablet for tablet 3e4921360ffc441593616b1f177aae21 (DEFAULT_TABLE table=test-table [id=41a6677b9c8a4fa3b9e93313ff207153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:47.524472 14365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e4921360ffc441593616b1f177aae21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:47.537395 14733 tablet_bootstrap.cc:492] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: Bootstrap starting.
I20260807 18:32:47.539862 14734 tablet_bootstrap.cc:492] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: Bootstrap starting.
I20260807 18:32:47.540940 14733 tablet_bootstrap.cc:654] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:47.542273 14733 log.cc:824] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:47.543496 14734 tablet_bootstrap.cc:654] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:47.543880 14735 tablet_bootstrap.cc:492] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: Bootstrap starting.
I20260807 18:32:47.544816 14734 log.cc:824] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:47.545850 14733 tablet_bootstrap.cc:492] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: No bootstrap required, opened a new log
I20260807 18:32:47.546142 14733 ts_tablet_manager.cc:1397] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260807 18:32:47.547874 14734 tablet_bootstrap.cc:492] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: No bootstrap required, opened a new log
I20260807 18:32:47.548166 14734 ts_tablet_manager.cc:1397] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260807 18:32:47.548192 14735 tablet_bootstrap.cc:654] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:47.549441 14735 log.cc:824] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:47.552502 14735 tablet_bootstrap.cc:492] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: No bootstrap required, opened a new log
I20260807 18:32:47.552754 14735 ts_tablet_manager.cc:1397] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260807 18:32:47.554308 14733 raft_consensus.cc:348] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00e7c1918fa4e67948670ccd382b006" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 37879 } } peers { permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 } } peers { permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 } }
I20260807 18:32:47.554765 14733 raft_consensus.cc:374] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:47.554908 14733 raft_consensus.cc:729] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d9a71fd2b04e60b630057d92bfaebd, State: Initialized, Role: FOLLOWER
I20260807 18:32:47.555656 14733 consensus_queue.cc:260] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00e7c1918fa4e67948670ccd382b006" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 37879 } } peers { permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 } } peers { permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 } }
I20260807 18:32:47.557247 14733 ts_tablet_manager.cc:1428] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260807 18:32:47.558458 14734 raft_consensus.cc:348] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00e7c1918fa4e67948670ccd382b006" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 37879 } } peers { permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 } } peers { permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 } }
I20260807 18:32:47.558856 14734 raft_consensus.cc:374] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:47.558967 14734 raft_consensus.cc:729] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8a51f1a0194690be7acb3579c3e303, State: Initialized, Role: FOLLOWER
I20260807 18:32:47.559631 14734 consensus_queue.cc:260] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00e7c1918fa4e67948670ccd382b006" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 37879 } } peers { permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 } } peers { permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 } }
I20260807 18:32:47.561305 14734 ts_tablet_manager.cc:1428] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260807 18:32:47.561746 14574 heartbeater.cc:503] Master 127.27.1.254:46479 was elected leader, sending a full tablet report...
I20260807 18:32:47.564374 14735 raft_consensus.cc:348] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00e7c1918fa4e67948670ccd382b006" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 37879 } } peers { permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 } } peers { permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 } }
I20260807 18:32:47.564728 14735 raft_consensus.cc:374] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:47.564862 14735 raft_consensus.cc:729] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00e7c1918fa4e67948670ccd382b006, State: Initialized, Role: FOLLOWER
I20260807 18:32:47.565438 14735 consensus_queue.cc:260] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00e7c1918fa4e67948670ccd382b006" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 37879 } } peers { permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 } } peers { permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 } }
I20260807 18:32:47.566457 14710 heartbeater.cc:503] Master 127.27.1.254:46479 was elected leader, sending a full tablet report...
I20260807 18:32:47.567070 14735 ts_tablet_manager.cc:1428] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
W20260807 18:32:47.687635 14712 tablet.cc:2369] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:32:47.718240 14575 tablet.cc:2369] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:32:47.735455 14435 tablet.cc:2369] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:32:47.750361 14741 raft_consensus.cc:482] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:32:47.750648 14741 raft_consensus.cc:504] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00e7c1918fa4e67948670ccd382b006" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 37879 } } peers { permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 } } peers { permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 } }
I20260807 18:32:47.752305 14741 leader_election.cc:288] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b8a51f1a0194690be7acb3579c3e303 (127.27.1.194:39977), 53d9a71fd2b04e60b630057d92bfaebd (127.27.1.193:44161)
I20260807 18:32:47.759073 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.752825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:55987 (local address 127.27.1.194:39977)
0807 18:32:47.753041 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:47.753047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:47.753146 (+ 99us) server_negotiation.cc:407] Connection header received
0807 18:32:47.753403 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:47.753411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:47.753511 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:47.753640 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:47.754140 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.755106 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.757084 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.758111 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.758267 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.758352 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:47.758654 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 18:32:47.758790 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:32:47.759207 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.752683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:39977 (local address 127.27.1.195:55987)
0807 18:32:47.753024 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:47.753048 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:47.753210 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:47.753707 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:47.753710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:47.753733 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:47.753974 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.753985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.755338 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.755343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.756911 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.756922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.758297 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.758303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.758477 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.758521 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 18:32:47.758622 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 18:32:47.758775 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":1}
I20260807 18:32:47.759652 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.752879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:50221 (local address 127.27.1.193:44161)
0807 18:32:47.753067 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:47.753074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:47.753796 (+ 722us) server_negotiation.cc:407] Connection header received
0807 18:32:47.753979 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:47.753995 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:47.754096 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:47.754180 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:47.754785 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.755737 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.757621 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.758648 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.758787 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.758848 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:47.759291 (+ 443us) server_negotiation.cc:299] Negotiation successful
0807 18:32:47.759486 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:32:47.760177 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.752873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44161 (local address 127.27.1.195:50221)
0807 18:32:47.753672 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:47.753693 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:47.753837 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:47.754223 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:47.754230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:47.754258 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:47.754609 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.754621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.755904 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.755907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.757426 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:47.757441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.758810 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.758816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:47.759049 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.759089 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 18:32:47.759840 (+ 751us) client_negotiation.cc:240] Negotiation successful
0807 18:32:47.759962 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":438,"threads_started":1}
I20260807 18:32:47.760213 14525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4921360ffc441593616b1f177aae21" candidate_uuid: "c00e7c1918fa4e67948670ccd382b006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b8a51f1a0194690be7acb3579c3e303" is_pre_election: true
I20260807 18:32:47.760809 14525 raft_consensus.cc:2454] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c00e7c1918fa4e67948670ccd382b006 in term 0.
I20260807 18:32:47.761209 14387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4921360ffc441593616b1f177aae21" candidate_uuid: "c00e7c1918fa4e67948670ccd382b006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53d9a71fd2b04e60b630057d92bfaebd" is_pre_election: true
I20260807 18:32:47.761467 14596 leader_election.cc:302] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b8a51f1a0194690be7acb3579c3e303, c00e7c1918fa4e67948670ccd382b006; no voters:
I20260807 18:32:47.761759 14387 raft_consensus.cc:2454] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c00e7c1918fa4e67948670ccd382b006 in term 0.
I20260807 18:32:47.761931 14741 raft_consensus.cc:2781] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:32:47.762086 14741 raft_consensus.cc:482] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:32:47.762220 14741 raft_consensus.cc:3037] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:47.764609 14741 raft_consensus.cc:504] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00e7c1918fa4e67948670ccd382b006" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 37879 } } peers { permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 } } peers { permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 } }
I20260807 18:32:47.765157 14741 leader_election.cc:288] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [CANDIDATE]: Term 1 election: Requested vote from peers 1b8a51f1a0194690be7acb3579c3e303 (127.27.1.194:39977), 53d9a71fd2b04e60b630057d92bfaebd (127.27.1.193:44161)
I20260807 18:32:47.765643 14525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4921360ffc441593616b1f177aae21" candidate_uuid: "c00e7c1918fa4e67948670ccd382b006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b8a51f1a0194690be7acb3579c3e303"
I20260807 18:32:47.765643 14387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4921360ffc441593616b1f177aae21" candidate_uuid: "c00e7c1918fa4e67948670ccd382b006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53d9a71fd2b04e60b630057d92bfaebd"
I20260807 18:32:47.765851 14387 raft_consensus.cc:3037] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:47.765851 14525 raft_consensus.cc:3037] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:47.769070 14387 raft_consensus.cc:2454] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c00e7c1918fa4e67948670ccd382b006 in term 1.
I20260807 18:32:47.769070 14525 raft_consensus.cc:2454] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c00e7c1918fa4e67948670ccd382b006 in term 1.
I20260807 18:32:47.769490 14596 leader_election.cc:302] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b8a51f1a0194690be7acb3579c3e303, c00e7c1918fa4e67948670ccd382b006; no voters:
I20260807 18:32:47.769794 14741 raft_consensus.cc:2781] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:32:47.770334 14741 raft_consensus.cc:686] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 1 LEADER]: Becoming Leader. State: Replica: c00e7c1918fa4e67948670ccd382b006, State: Running, Role: LEADER
I20260807 18:32:47.770804 14741 consensus_queue.cc:237] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00e7c1918fa4e67948670ccd382b006" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 37879 } } peers { permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 } } peers { permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 } }
I20260807 18:32:47.775509 14240 catalog_manager.cc:5495] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 reported cstate change: term changed from 0 to 1, leader changed from <none> to c00e7c1918fa4e67948670ccd382b006 (127.27.1.195). New cstate: current_term: 1 leader_uuid: "c00e7c1918fa4e67948670ccd382b006" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00e7c1918fa4e67948670ccd382b006" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 37879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 } health_report { overall_health: UNKNOWN } } }
I20260807 18:32:47.803905 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.798074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.27.1.195:37879)
0807 18:32:47.798276 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:47.798283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:47.798398 (+ 115us) server_negotiation.cc:407] Connection header received
0807 18:32:47.798672 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:47.798677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:47.798758 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:47.798924 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:32:47.799374 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.800135 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.801584 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.802004 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.802146 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.802703 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:32:47.803290 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:32:47.803458 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:47.803553 (+ 95us) server_negotiation.cc:299] Negotiation successful
0807 18:32:47.803632 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:32:47.812175 14525 raft_consensus.cc:1261] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303 [term 1 FOLLOWER]: Refusing update from remote peer c00e7c1918fa4e67948670ccd382b006: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:32:47.812178 14387 raft_consensus.cc:1261] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd [term 1 FOLLOWER]: Refusing update from remote peer c00e7c1918fa4e67948670ccd382b006: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:32:47.813252 14743 consensus_queue.cc:1035] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8a51f1a0194690be7acb3579c3e303" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 39977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:32:47.813660 14741 consensus_queue.cc:1035] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d9a71fd2b04e60b630057d92bfaebd" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:32:47.827673 14746 mvcc.cc:205] Tried to move back new op lower bound from 7315978517747093504 to 7315978517592936448. Current Snapshot: MvccSnapshot[applied={T|T < 7315978517747093504}]
I20260807 18:32:47.842005 14241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:32:47.842311 14241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:32:47.846143 14241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6eebd303cedf4248b70aee1fcf3fd12d: Sending DeleteTablet for 3 replicas of tablet 3e4921360ffc441593616b1f177aae21
I20260807 18:32:47.847189 14642 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e4921360ffc441593616b1f177aae21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:47 UTC) from {username='slave'} at 127.0.0.1:59308
I20260807 18:32:47.847342 14504 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e4921360ffc441593616b1f177aae21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:47 UTC) from {username='slave'} at 127.0.0.1:56588
I20260807 18:32:47.847594 14365 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e4921360ffc441593616b1f177aae21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:47 UTC) from {username='slave'} at 127.0.0.1:37604
I20260807 18:32:47.848773 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 14301
I20260807 18:32:47.852169 14759 tablet_replica.cc:330] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: stopping tablet replica
I20260807 18:32:47.852748 14759 raft_consensus.cc:2229] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:32:47.853400 14759 raft_consensus.cc:2258] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:47.854581 14758 tablet_replica.cc:330] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: stopping tablet replica
I20260807 18:32:47.855160 14758 raft_consensus.cc:2229] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:32:47.855742 14758 raft_consensus.cc:2258] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:47.856221 14759 ts_tablet_manager.cc:1905] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:47.857892 14758 ts_tablet_manager.cc:1905] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:47.863911 14757 tablet_replica.cc:330] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: stopping tablet replica
I20260807 18:32:47.864328 14757 raft_consensus.cc:2229] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:32:47.864622 14759 ts_tablet_manager.cc:1918] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:47.864851 14759 log.cc:1196] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/3e4921360ffc441593616b1f177aae21
I20260807 18:32:47.864995 14757 raft_consensus.cc:2258] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:47.865557 14759 ts_tablet_manager.cc:1939] T 3e4921360ffc441593616b1f177aae21 P 53d9a71fd2b04e60b630057d92bfaebd: Deleting consensus metadata
I20260807 18:32:47.867254 14757 ts_tablet_manager.cc:1905] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:47.867516 14225 catalog_manager.cc:4841] TS 53d9a71fd2b04e60b630057d92bfaebd (127.27.1.193:44161): tablet 3e4921360ffc441593616b1f177aae21 (table test-table [id=41a6677b9c8a4fa3b9e93313ff207153]) successfully deleted
I20260807 18:32:47.865279 14758 ts_tablet_manager.cc:1918] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:47.868312 14758 log.cc:1196] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/3e4921360ffc441593616b1f177aae21
I20260807 18:32:47.868897 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.850085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.27.1.193:44161)
0807 18:32:47.850303 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:47.850312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:47.859945 (+ 9633us) server_negotiation.cc:407] Connection header received
0807 18:32:47.860367 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:47.860372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:47.860470 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:47.860542 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:47.861235 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.862453 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.864813 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.865545 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.865650 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.867163 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:47.867175 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:47.867190 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:47.867912 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:47.867969 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:47.868008 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:47.868125 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:47.868346 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 18:32:47.868466 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:32:47.868974 14758 ts_tablet_manager.cc:1939] T 3e4921360ffc441593616b1f177aae21 P 1b8a51f1a0194690be7acb3579c3e303: Deleting consensus metadata
I20260807 18:32:47.870285 14429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:47.870944 14227 catalog_manager.cc:4841] TS 1b8a51f1a0194690be7acb3579c3e303 (127.27.1.194:39977): tablet 3e4921360ffc441593616b1f177aae21 (table test-table [id=41a6677b9c8a4fa3b9e93313ff207153]) successfully deleted
I20260807 18:32:47.876084 14757 ts_tablet_manager.cc:1918] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:47.876322 14757 log.cc:1196] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/3e4921360ffc441593616b1f177aae21
I20260807 18:32:47.876863 14757 ts_tablet_manager.cc:1939] T 3e4921360ffc441593616b1f177aae21 P c00e7c1918fa4e67948670ccd382b006: Deleting consensus metadata
I20260807 18:32:47.878674 14225 catalog_manager.cc:4841] TS c00e7c1918fa4e67948670ccd382b006 (127.27.1.195:37879): tablet 3e4921360ffc441593616b1f177aae21 (table test-table [id=41a6677b9c8a4fa3b9e93313ff207153]) successfully deleted
I20260807 18:32:47.964740 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 14301
I20260807 18:32:47.988889 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 14438
I20260807 18:32:47.998605 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:47.989832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.27.1.194:39977)
0807 18:32:47.990057 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:47.990064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:47.990233 (+ 169us) server_negotiation.cc:407] Connection header received
0807 18:32:47.990441 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:47.990447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:47.990571 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:47.990687 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:47.991232 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.992483 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.994502 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:47.995319 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:47.995446 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:47.996933 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:47.996946 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:47.996966 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:47.997723 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:47.997785 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:47.997823 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:47.997948 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:47.998124 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 18:32:47.998233 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 18:32:47.999397 14569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:48.088128 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 14438
I20260807 18:32:48.114741 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 14577
I20260807 18:32:48.121883 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:48.115661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.27.1.195:37879)
0807 18:32:48.115870 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:48.115878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:48.115903 (+ 25us) server_negotiation.cc:407] Connection header received
0807 18:32:48.115968 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:48.115973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:48.116069 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:48.116190 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:48.116622 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:48.117415 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:48.118670 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:48.119217 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:48.119353 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:48.120518 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:48.120527 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:48.120540 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:48.121107 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:48.121153 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:48.121174 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:48.121304 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:48.121437 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 18:32:48.121535 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 18:32:48.122692 14705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:48.241457 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 14577
I20260807 18:32:48.264133 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 14209
I20260807 18:32:48.265297 14272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:48.377005 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 14209
2026-08-07T18:32:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3045 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:32:48 dist-test-slave-1xrh krb5kdc[14767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:32:48 dist-test-slave-1xrh krb5kdc[14767](info): set up 2 sockets
Aug 07 18:32:48 dist-test-slave-1xrh krb5kdc[14767](info): commencing operation
krb5kdc: starting...
W20260807 18:32:49.573206 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:32:49 dist-test-slave-1xrh krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:32:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:32:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:32:49.878046 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:36439
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:36439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:50.217685 14781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:50.217971 14781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:50.218065 14781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:50.225742 14781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:32:50.225973 14781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:50.226058 14781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:50.226125 14781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:32:50.226171 14781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:32:50.236260 14781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:36439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:36439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:50.237535 14781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:50.238782 14781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:50.250532 14786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:50.250469 14789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:50.250507 14787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:50.251780 14781 server_base.cc:1034] running on GCE node
I20260807 18:32:50.252542 14781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:50.253988 14781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:50.255239 14781 hybrid_clock.cc:648] HybridClock initialized: now 1786127570255196 us; error 24 us; skew 500 ppm
I20260807 18:32:50.255596 14781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:50 dist-test-slave-1xrh krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127570, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:50.258741 14781 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:32:50.260149 14781 webserver.cc:460] Webserver started at http://127.27.1.254:40019/ using document root <none> and password file <none>
I20260807 18:32:50.260772 14781 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:50.260895 14781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:50.261153 14781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:50.264120 14781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "284269a59ab54f40b645b073a1a32e7c"
format_stamp: "Formatted at 2026-08-07 18:32:50 on dist-test-slave-1xrh"
server_key: "a7ffad0778812751683b80d3cbeb24c1"
server_key_iv: "0b0c1434c177aabdde68c1f15ceef813"
server_key_version: "encryptionkey@0"
I20260807 18:32:50.264822 14781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "284269a59ab54f40b645b073a1a32e7c"
format_stamp: "Formatted at 2026-08-07 18:32:50 on dist-test-slave-1xrh"
server_key: "a7ffad0778812751683b80d3cbeb24c1"
server_key_iv: "0b0c1434c177aabdde68c1f15ceef813"
server_key_version: "encryptionkey@0"
I20260807 18:32:50.269773 14781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:32:50.273156 14796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:50.274415 14781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:32:50.274595 14781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "284269a59ab54f40b645b073a1a32e7c"
format_stamp: "Formatted at 2026-08-07 18:32:50 on dist-test-slave-1xrh"
server_key: "a7ffad0778812751683b80d3cbeb24c1"
server_key_iv: "0b0c1434c177aabdde68c1f15ceef813"
server_key_version: "encryptionkey@0"
I20260807 18:32:50.274802 14781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:50.289939 14781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:50.293741 14781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:50.294034 14781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:50.312522 14781 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:36439
I20260807 18:32:50.312565 14851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:36439 every 8 connection(s)
I20260807 18:32:50.313885 14781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:32:50.316744 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 14781
I20260807 18:32:50.316942 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:32:50.317423 27655 external_mini_cluster.cc:1428] Setting key 8dd5872d52ab0d7b4211aaf9e1c10eeb
I20260807 18:32:50.317669 14852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:50.332396 14852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c: Bootstrap starting.
Aug 07 18:32:50 dist-test-slave-1xrh krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:50.335600 14852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:50.336664 14852 log.cc:824] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:50.339572 14852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c: No bootstrap required, opened a new log
I20260807 18:32:50.340941 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:50.319264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.27.1.254:36439)
0807 18:32:50.320041 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:50.320052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:50.320098 (+ 46us) server_negotiation.cc:407] Connection header received
0807 18:32:50.322198 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:50.322219 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:50.322578 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:50.322874 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:50.324824 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:50.326078 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:50.328004 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:50.328798 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:50.328929 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:50.333873 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:50.333881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:50.333894 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:50.333928 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:50.337095 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:50.337777 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:50.337784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:50.337789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:50.337940 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:50.338327 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:50.338336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:50.338338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:50.338836 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:50.339137 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:50.339636 (+ 499us) server_negotiation.cc:299] Negotiation successful
0807 18:32:50.340037 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":373,"threads_started":1}
I20260807 18:32:50.348027 14852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284269a59ab54f40b645b073a1a32e7c" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 36439 } }
I20260807 18:32:50.348470 14852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:50.348589 14852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284269a59ab54f40b645b073a1a32e7c, State: Initialized, Role: FOLLOWER
I20260807 18:32:50.349134 14852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284269a59ab54f40b645b073a1a32e7c" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 36439 } }
I20260807 18:32:50.349385 14852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:32:50.349538 14852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:32:50.349711 14852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:50.352033 14852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284269a59ab54f40b645b073a1a32e7c" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 36439 } }
I20260807 18:32:50.352584 14852 leader_election.cc:302] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 284269a59ab54f40b645b073a1a32e7c; no voters:
I20260807 18:32:50.353348 14852 leader_election.cc:288] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:32:50.353412 14858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:32:50.354226 14858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [term 1 LEADER]: Becoming Leader. State: Replica: 284269a59ab54f40b645b073a1a32e7c, State: Running, Role: LEADER
I20260807 18:32:50.354775 14858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284269a59ab54f40b645b073a1a32e7c" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 36439 } }
I20260807 18:32:50.355628 14852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:32:50.357554 14859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "284269a59ab54f40b645b073a1a32e7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284269a59ab54f40b645b073a1a32e7c" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 36439 } } }
I20260807 18:32:50.359246 14859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:50.358265 14860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 284269a59ab54f40b645b073a1a32e7c. Latest consensus state: current_term: 1 leader_uuid: "284269a59ab54f40b645b073a1a32e7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284269a59ab54f40b645b073a1a32e7c" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 36439 } } }
I20260807 18:32:50.359684 14860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:50.360773 14866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:32:50.365773 14866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:32:50.374050 14866 catalog_manager.cc:1347] Generated new cluster ID: 05c2ab4b7b9249c68b6c51f841946cfb
I20260807 18:32:50.374178 14866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:32:50.414393 14866 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:32:50.415789 14866 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:32:50.425196 14866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c: Generated new TSK 0
I20260807 18:32:50.426162 14866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:32:50.504092 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:36439
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:50.842716 14879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:50.843012 14879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:50.843112 14879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:50.850286 14879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:50.850386 14879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:50.850495 14879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:32:50.860510 14879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:36439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:50.861812 14879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:50.863093 14879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:50.876412 14885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:50.876384 14886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:50.876586 14888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:50.877159 14879 server_base.cc:1034] running on GCE node
I20260807 18:32:50.877774 14879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:50.878772 14879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:50.880002 14879 hybrid_clock.cc:648] HybridClock initialized: now 1786127570879939 us; error 47 us; skew 500 ppm
I20260807 18:32:50.880322 14879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:50 dist-test-slave-1xrh krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127570, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:50.889153 14879 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:32:50.890599 14879 webserver.cc:460] Webserver started at http://127.27.1.193:35623/ using document root <none> and password file <none>
I20260807 18:32:50.891302 14879 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:50.891409 14879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:50.891669 14879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:50.894449 14879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "70e59774f3db4472aea8d5754951a4ed"
format_stamp: "Formatted at 2026-08-07 18:32:50 on dist-test-slave-1xrh"
server_key: "f880aad9df899d1d5598d320767bd1a7"
server_key_iv: "2f9388dd5181f8076ca5640ca5efa394"
server_key_version: "encryptionkey@0"
I20260807 18:32:50.895212 14879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "70e59774f3db4472aea8d5754951a4ed"
format_stamp: "Formatted at 2026-08-07 18:32:50 on dist-test-slave-1xrh"
server_key: "f880aad9df899d1d5598d320767bd1a7"
server_key_iv: "2f9388dd5181f8076ca5640ca5efa394"
server_key_version: "encryptionkey@0"
I20260807 18:32:50.900130 14879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 18:32:50.903390 14895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:50.904641 14879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 18:32:50.904773 14879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "70e59774f3db4472aea8d5754951a4ed"
format_stamp: "Formatted at 2026-08-07 18:32:50 on dist-test-slave-1xrh"
server_key: "f880aad9df899d1d5598d320767bd1a7"
server_key_iv: "2f9388dd5181f8076ca5640ca5efa394"
server_key_version: "encryptionkey@0"
I20260807 18:32:50.904927 14879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:50.918820 14879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:50.922566 14879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:50.922863 14879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:50.923911 14879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:50.925563 14879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:50.925689 14879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:50.925846 14879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:50.925925 14879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:50.958928 14879 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:42325
I20260807 18:32:50.958971 15012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:42325 every 8 connection(s)
I20260807 18:32:50.960256 14879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:32:50.963315 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 14879
I20260807 18:32:50.963541 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:32:50.964011 27655 external_mini_cluster.cc:1428] Setting key d2aa80f3f5a3b7377fb2f90a5c51fb8d
Aug 07 18:32:50 dist-test-slave-1xrh krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127570, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:50.981679 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:50.962966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:40055 (local address 127.27.1.254:36439)
0807 18:32:50.963487 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:50.963493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:50.964562 (+ 1069us) server_negotiation.cc:407] Connection header received
0807 18:32:50.966323 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:50.966329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:50.966425 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:50.966559 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:50.968048 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:50.969214 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:50.970736 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:50.971441 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:50.971603 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:50.974783 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:50.974792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:50.974800 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:50.974845 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:50.978416 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:50.979330 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:50.979334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:50.979336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:50.979499 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:50.980004 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:50.980008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:50.980009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:50.980268 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:50.980379 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:50.981244 (+ 865us) server_negotiation.cc:299] Negotiation successful
0807 18:32:50.981408 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":270,"threads_started":1}
I20260807 18:32:50.983160 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:50.963504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:36439 (local address 127.27.1.193:40055)
0807 18:32:50.964392 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:50.964438 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:50.966068 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:50.966742 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:50.966749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:50.967327 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:50.967865 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:50.967882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:50.969375 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:50.969379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:50.970585 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:50.970597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:50.971652 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:50.971657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:50.971840 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:50.972519 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:50.972540 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:50.974573 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:50.978577 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:50.978585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:50.978603 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:50.979215 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:50.979628 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:50.979635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:50.979640 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:50.979890 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:50.980400 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:50.980410 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:50.980836 (+ 426us) client_negotiation.cc:769] Sending connection context
0807 18:32:50.981141 (+ 305us) client_negotiation.cc:240] Negotiation successful
0807 18:32:50.981686 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":383,"threads_started":1}
I20260807 18:32:50.984764 15013 heartbeater.cc:347] Connected to a master server at 127.27.1.254:36439
I20260807 18:32:50.985093 15013 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:50.985816 15013 heartbeater.cc:511] Master 127.27.1.254:36439 requested a full tablet report, sending...
I20260807 18:32:50.987947 14815 ts_manager.cc:194] Registered new tserver with Master: 70e59774f3db4472aea8d5754951a4ed (127.27.1.193:42325)
I20260807 18:32:50.989369 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:40055
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:32:51.034183 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:36439
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:51.371399 15019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:51.371726 15019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:51.371898 15019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:51.379060 15019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:51.379215 15019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:51.379382 15019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:32:51.388973 15019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:36439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:51.390252 15019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:51.391600 15019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:51.404848 15024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:51.404938 15025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:51.404958 15027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:51.406041 15019 server_base.cc:1034] running on GCE node
I20260807 18:32:51.406651 15019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:51.407708 15019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:51.408931 15019 hybrid_clock.cc:648] HybridClock initialized: now 1786127571408895 us; error 51 us; skew 500 ppm
I20260807 18:32:51.409240 15019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:51 dist-test-slave-1xrh krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127571, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:51.412357 15019 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:32:51.413779 15019 webserver.cc:460] Webserver started at http://127.27.1.194:46033/ using document root <none> and password file <none>
I20260807 18:32:51.414474 15019 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:51.414577 15019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:51.414829 15019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:51.417668 15019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "e6f94ad6633846f2a564486fb3fd1c68"
format_stamp: "Formatted at 2026-08-07 18:32:51 on dist-test-slave-1xrh"
server_key: "c874dd20a44ec46156c3d4fc293f0b28"
server_key_iv: "f92b59fdcdaa92bec99a60e5774b5576"
server_key_version: "encryptionkey@0"
I20260807 18:32:51.418360 15019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "e6f94ad6633846f2a564486fb3fd1c68"
format_stamp: "Formatted at 2026-08-07 18:32:51 on dist-test-slave-1xrh"
server_key: "c874dd20a44ec46156c3d4fc293f0b28"
server_key_iv: "f92b59fdcdaa92bec99a60e5774b5576"
server_key_version: "encryptionkey@0"
I20260807 18:32:51.423359 15019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 18:32:51.426702 15034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:51.428007 15019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:32:51.428136 15019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "e6f94ad6633846f2a564486fb3fd1c68"
format_stamp: "Formatted at 2026-08-07 18:32:51 on dist-test-slave-1xrh"
server_key: "c874dd20a44ec46156c3d4fc293f0b28"
server_key_iv: "f92b59fdcdaa92bec99a60e5774b5576"
server_key_version: "encryptionkey@0"
I20260807 18:32:51.428295 15019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:51.454860 15019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:51.459012 15019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:51.459322 15019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:51.460245 15019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:51.461575 15019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:51.461647 15019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:51.461730 15019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:51.461802 15019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:51.495051 15019 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:46173
I20260807 18:32:51.495088 15152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:46173 every 8 connection(s)
I20260807 18:32:51.496410 15019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:32:51.505941 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 15019
I20260807 18:32:51.506138 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:32:51.506593 27655 external_mini_cluster.cc:1428] Setting key e25ef70a8e64ee4b7ce9fed603152102
Aug 07 18:32:51 dist-test-slave-1xrh krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127571, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:51.516592 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:51.498868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:36519 (local address 127.27.1.254:36439)
0807 18:32:51.499377 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:51.499382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:51.500223 (+ 841us) server_negotiation.cc:407] Connection header received
0807 18:32:51.502049 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:51.502055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:51.502158 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:51.502283 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:51.503858 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:51.504597 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:51.506177 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:51.506706 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:51.506860 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:51.510740 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:51.510745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:51.510748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:51.510784 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:51.513446 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:51.514209 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:51.514219 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:51.514222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:51.514356 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:51.514804 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:51.514810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:51.514813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:51.515074 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:51.515284 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:51.515966 (+ 682us) server_negotiation.cc:299] Negotiation successful
0807 18:32:51.516192 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":301,"threads_started":1}
I20260807 18:32:51.517541 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:51.499333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:36439 (local address 127.27.1.194:36519)
0807 18:32:51.500043 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:51.500087 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:51.501764 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:51.502521 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:51.502531 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:51.503036 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:51.503667 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:51.503685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:51.504765 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:51.504770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:51.505983 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:51.506038 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:51.506949 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:51.506956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:51.507324 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:51.508046 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:51.508074 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:51.510508 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:51.513618 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:51.513627 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:51.513643 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:51.514071 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:51.514480 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:51.514485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:51.514489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:51.514676 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:51.515276 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:51.515283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:51.515605 (+ 322us) client_negotiation.cc:769] Sending connection context
0807 18:32:51.515871 (+ 266us) client_negotiation.cc:240] Negotiation successful
0807 18:32:51.516401 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":300,"threads_started":1}
I20260807 18:32:51.519078 15153 heartbeater.cc:347] Connected to a master server at 127.27.1.254:36439
I20260807 18:32:51.519419 15153 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:51.520131 15153 heartbeater.cc:511] Master 127.27.1.254:36439 requested a full tablet report, sending...
I20260807 18:32:51.521623 14815 ts_manager.cc:194] Registered new tserver with Master: e6f94ad6633846f2a564486fb3fd1c68 (127.27.1.194:46173)
I20260807 18:32:51.522466 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:36519
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:32:51.574918 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:36439
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:51.919098 15159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:51.919445 15159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:51.919612 15159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:51.926847 15159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:51.927000 15159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:51.927196 15159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:32:51.937057 15159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:36439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:51.938409 15159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:51.939962 15159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:51.953179 15165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:51.953220 15167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:51.953608 15164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:51.954150 15159 server_base.cc:1034] running on GCE node
I20260807 18:32:51.954708 15159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:51.955725 15159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:51.957067 15159 hybrid_clock.cc:648] HybridClock initialized: now 1786127571957029 us; error 26 us; skew 500 ppm
I20260807 18:32:51.957428 15159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:51 dist-test-slave-1xrh krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127571, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:51.960848 15159 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:32:51.962546 15159 webserver.cc:460] Webserver started at http://127.27.1.195:40647/ using document root <none> and password file <none>
I20260807 18:32:51.963311 15159 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:51.963469 15159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:51.963809 15159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:51.966641 15159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "bc8b72b593aa4eb5a3564c45c1c72237"
format_stamp: "Formatted at 2026-08-07 18:32:51 on dist-test-slave-1xrh"
server_key: "f1724d90fa8c5c7079c7da61469ee8a1"
server_key_iv: "2b83cba9543c517504be369a1bd76fa1"
server_key_version: "encryptionkey@0"
I20260807 18:32:51.967489 15159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "bc8b72b593aa4eb5a3564c45c1c72237"
format_stamp: "Formatted at 2026-08-07 18:32:51 on dist-test-slave-1xrh"
server_key: "f1724d90fa8c5c7079c7da61469ee8a1"
server_key_iv: "2b83cba9543c517504be369a1bd76fa1"
server_key_version: "encryptionkey@0"
I20260807 18:32:51.972615 15159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 18:32:51.976066 15174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:51.977370 15159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 18:32:51.977584 15159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "bc8b72b593aa4eb5a3564c45c1c72237"
format_stamp: "Formatted at 2026-08-07 18:32:51 on dist-test-slave-1xrh"
server_key: "f1724d90fa8c5c7079c7da61469ee8a1"
server_key_iv: "2b83cba9543c517504be369a1bd76fa1"
server_key_version: "encryptionkey@0"
I20260807 18:32:51.977866 15159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:51.992352 15013 heartbeater.cc:503] Master 127.27.1.254:36439 was elected leader, sending a full tablet report...
I20260807 18:32:52.020709 15159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:52.024861 15159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:52.025115 15159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:52.026015 15159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:52.027384 15159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:52.027467 15159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:52.027563 15159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:52.027607 15159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:52.061151 15159 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:44969
I20260807 18:32:52.061187 15292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:44969 every 8 connection(s)
I20260807 18:32:52.062505 15159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:32:52.064404 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 15159
I20260807 18:32:52.064611 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:32:52.065083 27655 external_mini_cluster.cc:1428] Setting key db5867bad0a6765a53edf04b6cb4c28b
Aug 07 18:32:52 dist-test-slave-1xrh krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127571, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:52.083207 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.065242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:36905 (local address 127.27.1.254:36439)
0807 18:32:52.065783 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.065789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.066975 (+ 1186us) server_negotiation.cc:407] Connection header received
0807 18:32:52.069058 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.069064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.069157 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.069283 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:52.070714 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.071607 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.073353 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.073743 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.073877 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.077216 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:52.077222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:52.077225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:52.077256 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:52.080193 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:52.080978 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:52.080981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:52.080983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:52.081107 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:52.081546 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:52.081549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:52.081551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:52.081843 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:52.082000 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.082669 (+ 669us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.082870 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":255,"threads_started":1}
I20260807 18:32:52.084118 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.065825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:36439 (local address 127.27.1.195:36905)
0807 18:32:52.066657 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:52.066702 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:52.068801 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:52.069518 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:52.069526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:52.070004 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:52.070546 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.070564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.071767 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.071771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.073169 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.073181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.073965 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.073974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.074219 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.074818 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:52.074839 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:52.077007 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:52.080351 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:52.080357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:52.080367 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:52.080887 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:52.081222 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:52.081226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:52.081229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:52.081449 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:52.081949 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:52.081954 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:52.082241 (+ 287us) client_negotiation.cc:769] Sending connection context
0807 18:32:52.082595 (+ 354us) client_negotiation.cc:240] Negotiation successful
0807 18:32:52.082949 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":362,"threads_started":1}
I20260807 18:32:52.085346 15293 heartbeater.cc:347] Connected to a master server at 127.27.1.254:36439
I20260807 18:32:52.085664 15293 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:52.086380 15293 heartbeater.cc:511] Master 127.27.1.254:36439 requested a full tablet report, sending...
I20260807 18:32:52.087764 14815 ts_manager.cc:194] Registered new tserver with Master: bc8b72b593aa4eb5a3564c45c1c72237 (127.27.1.195:44969)
I20260807 18:32:52.088501 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:36905
I20260807 18:32:52.092471 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:32:52.109928 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.096490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.27.1.254:36439)
0807 18:32:52.096635 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.096639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.097130 (+ 491us) server_negotiation.cc:407] Connection header received
0807 18:32:52.097325 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.097330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.097399 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.097501 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:52.098998 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.099777 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.101148 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.101587 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.101719 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.104551 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:52.104558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:52.104562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:52.104600 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:52.106977 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:52.107735 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:52.107740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:52.107744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:52.107867 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:52.108402 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:52.108409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:52.108411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:52.108773 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:52.109047 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.109414 (+ 367us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.109596 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 18:32:52.113759 14815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:32:52.120539 14815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:32:52.151546 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.132932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.27.1.194:46173)
0807 18:32:52.133635 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.133644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.140566 (+ 6922us) server_negotiation.cc:407] Connection header received
0807 18:32:52.140711 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.140722 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.141058 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.142997 (+ 1939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:52.143346 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.145093 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.148831 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.150330 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.150474 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.150687 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.151024 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.151222 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":326,"threads_started":1}
I20260807 18:32:52.151674 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.132333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:46173 (local address 127.0.0.1:39370)
0807 18:32:52.134086 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:52.134147 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:52.134374 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:52.142087 (+ 7713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:52.142098 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:52.142141 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:52.142711 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.142728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.146025 (+ 3297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.146030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.148647 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.148662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.150560 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.150566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.150776 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.150823 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 18:32:52.150961 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 18:32:52.151191 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1431,"mutex_wait_us":1,"thread_start_us":1265,"threads_started":1}
I20260807 18:32:52.154162 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.132446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:42325 (local address 127.0.0.1:47836)
0807 18:32:52.144451 (+ 12005us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:52.144472 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:52.144595 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:52.146042 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:52.146047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:52.146082 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:52.146570 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.146581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.148902 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.148907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.151161 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.151176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.153315 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.153323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.153490 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.153522 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 18:32:52.153616 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 18:32:52.153745 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11883,"mutex_wait_us":74,"thread_start_us":12986,"threads_started":1}
I20260807 18:32:52.154523 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.133036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.27.1.193:42325)
0807 18:32:52.145340 (+ 12304us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.145353 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.145386 (+ 33us) server_negotiation.cc:407] Connection header received
0807 18:32:52.145512 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.145522 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.145809 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.146784 (+ 975us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:52.147632 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.148722 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.151337 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.152777 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.152932 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.153177 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.153988 (+ 811us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.154156 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12088,"thread_start_us":11920,"threads_started":1}
I20260807 18:32:52.154860 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.134571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.27.1.195:44969)
0807 18:32:52.146470 (+ 11899us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.146478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.146509 (+ 31us) server_negotiation.cc:407] Connection header received
0807 18:32:52.146609 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.146615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.146823 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.147065 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:52.147861 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.149381 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.152172 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.153514 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.153661 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.153879 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.154282 (+ 403us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.154436 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11719,"thread_start_us":12124,"threads_started":1}
I20260807 18:32:52.154808 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.134265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:44969 (local address 127.0.0.1:55748)
0807 18:32:52.145684 (+ 11419us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:52.145702 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:52.145812 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:52.147053 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:52.147058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:52.147083 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:52.147652 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.147664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.149562 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.149567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.151767 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.151781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.153721 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.153727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.153901 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.153940 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 18:32:52.154041 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 18:32:52.154169 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11299,"mutex_wait_us":30,"thread_start_us":10396,"threads_started":1}
I20260807 18:32:52.156128 15085 tablet_service.cc:1467] Processing CreateTablet for tablet a948eefab9cf42d99b0ecff679ec80e9 (DEFAULT_TABLE table=test-table [id=284662a0cc4c41a2b9e0e1d7a633c697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:52.157641 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a948eefab9cf42d99b0ecff679ec80e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:52.158406 15225 tablet_service.cc:1467] Processing CreateTablet for tablet a948eefab9cf42d99b0ecff679ec80e9 (DEFAULT_TABLE table=test-table [id=284662a0cc4c41a2b9e0e1d7a633c697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:52.158413 14946 tablet_service.cc:1467] Processing CreateTablet for tablet a948eefab9cf42d99b0ecff679ec80e9 (DEFAULT_TABLE table=test-table [id=284662a0cc4c41a2b9e0e1d7a633c697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:52.159823 15225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a948eefab9cf42d99b0ecff679ec80e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:52.159824 14946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a948eefab9cf42d99b0ecff679ec80e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:52.169266 15314 tablet_bootstrap.cc:492] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: Bootstrap starting.
I20260807 18:32:52.172395 15314 tablet_bootstrap.cc:654] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:52.173394 15314 log.cc:824] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:52.176079 15317 tablet_bootstrap.cc:492] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: Bootstrap starting.
I20260807 18:32:52.176656 15314 tablet_bootstrap.cc:492] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: No bootstrap required, opened a new log
I20260807 18:32:52.176926 15314 ts_tablet_manager.cc:1397] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 18:32:52.179562 15317 tablet_bootstrap.cc:654] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:52.180352 15316 tablet_bootstrap.cc:492] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: Bootstrap starting.
I20260807 18:32:52.180787 15317 log.cc:824] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:52.183988 15317 tablet_bootstrap.cc:492] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: No bootstrap required, opened a new log
I20260807 18:32:52.184335 15317 ts_tablet_manager.cc:1397] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260807 18:32:52.184651 15316 tablet_bootstrap.cc:654] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:52.185833 15316 log.cc:824] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:52.187598 15314 raft_consensus.cc:348] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 } } peers { permanent_uuid: "70e59774f3db4472aea8d5754951a4ed" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 42325 } } peers { permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 } }
I20260807 18:32:52.188045 15314 raft_consensus.cc:374] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:52.188196 15314 raft_consensus.cc:729] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f94ad6633846f2a564486fb3fd1c68, State: Initialized, Role: FOLLOWER
I20260807 18:32:52.189013 15316 tablet_bootstrap.cc:492] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: No bootstrap required, opened a new log
I20260807 18:32:52.188858 15314 consensus_queue.cc:260] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 } } peers { permanent_uuid: "70e59774f3db4472aea8d5754951a4ed" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 42325 } } peers { permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 } }
I20260807 18:32:52.189291 15316 ts_tablet_manager.cc:1397] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 18:32:52.189956 15153 heartbeater.cc:503] Master 127.27.1.254:36439 was elected leader, sending a full tablet report...
I20260807 18:32:52.191985 15314 ts_tablet_manager.cc:1428] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260807 18:32:52.192988 15317 raft_consensus.cc:348] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 } } peers { permanent_uuid: "70e59774f3db4472aea8d5754951a4ed" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 42325 } } peers { permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 } }
I20260807 18:32:52.193401 15317 raft_consensus.cc:374] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:52.193517 15317 raft_consensus.cc:729] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8b72b593aa4eb5a3564c45c1c72237, State: Initialized, Role: FOLLOWER
I20260807 18:32:52.194332 15317 consensus_queue.cc:260] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 } } peers { permanent_uuid: "70e59774f3db4472aea8d5754951a4ed" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 42325 } } peers { permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 } }
I20260807 18:32:52.198189 15316 raft_consensus.cc:348] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 } } peers { permanent_uuid: "70e59774f3db4472aea8d5754951a4ed" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 42325 } } peers { permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 } }
I20260807 18:32:52.198532 15316 raft_consensus.cc:374] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:52.198648 15316 raft_consensus.cc:729] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e59774f3db4472aea8d5754951a4ed, State: Initialized, Role: FOLLOWER
I20260807 18:32:52.199196 15316 consensus_queue.cc:260] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 } } peers { permanent_uuid: "70e59774f3db4472aea8d5754951a4ed" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 42325 } } peers { permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 } }
I20260807 18:32:52.200695 15316 ts_tablet_manager.cc:1428] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260807 18:32:52.201458 15293 heartbeater.cc:503] Master 127.27.1.254:36439 was elected leader, sending a full tablet report...
I20260807 18:32:52.202061 15317 ts_tablet_manager.cc:1428] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: Time spent starting tablet: real 0.018s user 0.010s sys 0.006s
W20260807 18:32:52.212474 15014 tablet.cc:2369] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:32:52.248364 15154 tablet.cc:2369] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:32:52.314314 15295 tablet.cc:2369] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:32:52.530810 15322 raft_consensus.cc:482] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:32:52.531206 15322 raft_consensus.cc:504] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 } } peers { permanent_uuid: "70e59774f3db4472aea8d5754951a4ed" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 42325 } } peers { permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 } }
I20260807 18:32:52.533480 15322 leader_election.cc:288] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6f94ad6633846f2a564486fb3fd1c68 (127.27.1.194:46173), bc8b72b593aa4eb5a3564c45c1c72237 (127.27.1.195:44969)
I20260807 18:32:52.542610 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.534479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:58243 (local address 127.27.1.194:46173)
0807 18:32:52.535380 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.535389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.535414 (+ 25us) server_negotiation.cc:407] Connection header received
0807 18:32:52.535502 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.535510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.535611 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.535722 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:52.536552 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.537679 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.540774 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.541648 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.541835 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.541930 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.542221 (+ 291us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.542347 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500}
I20260807 18:32:52.542717 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.533838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:46173 (local address 127.27.1.193:58243)
0807 18:32:52.534550 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:52.534574 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:52.534729 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:52.535816 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:52.535821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:52.535850 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:52.536263 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.536278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.537843 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.537851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.540580 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.540594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.541859 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.541866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.542053 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.542078 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 18:32:52.542194 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 18:32:52.542327 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":144,"thread_start_us":321,"threads_started":1}
I20260807 18:32:52.543882 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.535015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:55423 (local address 127.27.1.195:44969)
0807 18:32:52.535305 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.535312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.535568 (+ 256us) server_negotiation.cc:407] Connection header received
0807 18:32:52.535837 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.535842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.535932 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.535995 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:52.536974 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.537969 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.540750 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.542362 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.542550 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.542643 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.543332 (+ 689us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.543537 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260807 18:32:52.544196 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a948eefab9cf42d99b0ecff679ec80e9" candidate_uuid: "70e59774f3db4472aea8d5754951a4ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f94ad6633846f2a564486fb3fd1c68" is_pre_election: true
I20260807 18:32:52.544740 15105 raft_consensus.cc:2454] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e59774f3db4472aea8d5754951a4ed in term 0.
I20260807 18:32:52.544813 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.534768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:44969 (local address 127.27.1.193:55423)
0807 18:32:52.535459 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:52.535476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:52.535674 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:52.536330 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:52.536333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:52.536354 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:52.536769 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.536782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.538130 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.538133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.540579 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:52.540594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.542838 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.542847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:52.543060 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.543102 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 18:32:52.544108 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0807 18:32:52.544311 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":94,"thread_start_us":740,"threads_started":1}
I20260807 18:32:52.545423 14898 leader_election.cc:302] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e59774f3db4472aea8d5754951a4ed, e6f94ad6633846f2a564486fb3fd1c68; no voters:
I20260807 18:32:52.545941 15322 raft_consensus.cc:2781] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:32:52.545913 15245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a948eefab9cf42d99b0ecff679ec80e9" candidate_uuid: "70e59774f3db4472aea8d5754951a4ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" is_pre_election: true
I20260807 18:32:52.546085 15322 raft_consensus.cc:482] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:32:52.546188 15322 raft_consensus.cc:3037] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:52.546415 15245 raft_consensus.cc:2454] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e59774f3db4472aea8d5754951a4ed in term 0.
I20260807 18:32:52.551389 15322 raft_consensus.cc:504] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 } } peers { permanent_uuid: "70e59774f3db4472aea8d5754951a4ed" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 42325 } } peers { permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 } }
I20260807 18:32:52.552218 15322 leader_election.cc:288] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [CANDIDATE]: Term 1 election: Requested vote from peers e6f94ad6633846f2a564486fb3fd1c68 (127.27.1.194:46173), bc8b72b593aa4eb5a3564c45c1c72237 (127.27.1.195:44969)
I20260807 18:32:52.552642 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a948eefab9cf42d99b0ecff679ec80e9" candidate_uuid: "70e59774f3db4472aea8d5754951a4ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f94ad6633846f2a564486fb3fd1c68"
I20260807 18:32:52.552759 15245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a948eefab9cf42d99b0ecff679ec80e9" candidate_uuid: "70e59774f3db4472aea8d5754951a4ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8b72b593aa4eb5a3564c45c1c72237"
I20260807 18:32:52.552858 15105 raft_consensus.cc:3037] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:52.553027 15245 raft_consensus.cc:3037] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:52.555435 15105 raft_consensus.cc:2454] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e59774f3db4472aea8d5754951a4ed in term 1.
I20260807 18:32:52.555998 14898 leader_election.cc:302] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e59774f3db4472aea8d5754951a4ed, e6f94ad6633846f2a564486fb3fd1c68; no voters:
I20260807 18:32:52.556077 15245 raft_consensus.cc:2454] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e59774f3db4472aea8d5754951a4ed in term 1.
I20260807 18:32:52.556588 15322 raft_consensus.cc:2781] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:32:52.557142 15322 raft_consensus.cc:686] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 1 LEADER]: Becoming Leader. State: Replica: 70e59774f3db4472aea8d5754951a4ed, State: Running, Role: LEADER
I20260807 18:32:52.557698 15322 consensus_queue.cc:237] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 } } peers { permanent_uuid: "70e59774f3db4472aea8d5754951a4ed" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 42325 } } peers { permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 } }
I20260807 18:32:52.563309 14815 catalog_manager.cc:5495] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 70e59774f3db4472aea8d5754951a4ed (127.27.1.193). New cstate: current_term: 1 leader_uuid: "70e59774f3db4472aea8d5754951a4ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70e59774f3db4472aea8d5754951a4ed" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 42325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 } health_report { overall_health: UNKNOWN } } }
I20260807 18:32:52.611048 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.605200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.27.1.193:42325)
0807 18:32:52.605417 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.605423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.605522 (+ 99us) server_negotiation.cc:407] Connection header received
0807 18:32:52.605761 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.605764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.605859 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.606019 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:32:52.606439 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.607273 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.608761 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.609165 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.609301 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.609900 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:32:52.610428 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:32:52.610618 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.610777 (+ 159us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.610825 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 18:32:52.618944 15245 raft_consensus.cc:1261] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237 [term 1 FOLLOWER]: Refusing update from remote peer 70e59774f3db4472aea8d5754951a4ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:32:52.619047 15105 raft_consensus.cc:1261] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68 [term 1 FOLLOWER]: Refusing update from remote peer 70e59774f3db4472aea8d5754951a4ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:32:52.619961 15325 consensus_queue.cc:1035] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc8b72b593aa4eb5a3564c45c1c72237" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 44969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:32:52.620307 15322 consensus_queue.cc:1035] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f94ad6633846f2a564486fb3fd1c68" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:32:52.630903 15328 mvcc.cc:205] Tried to move back new op lower bound from 7315978537437876224 to 7315978537201168384. Current Snapshot: MvccSnapshot[applied={T|T < 7315978537437876224}]
I20260807 18:32:52.644591 14815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:32:52.644800 14815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:32:52.648322 14815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 284269a59ab54f40b645b073a1a32e7c: Sending DeleteTablet for 3 replicas of tablet a948eefab9cf42d99b0ecff679ec80e9
I20260807 18:32:52.649523 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet a948eefab9cf42d99b0ecff679ec80e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:52 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:39370
I20260807 18:32:52.649825 15225 tablet_service.cc:1514] Processing DeleteTablet for tablet a948eefab9cf42d99b0ecff679ec80e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:52 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:55748
I20260807 18:32:52.650246 14946 tablet_service.cc:1514] Processing DeleteTablet for tablet a948eefab9cf42d99b0ecff679ec80e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:52 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:47836
I20260807 18:32:52.651576 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 14879
I20260807 18:32:52.665364 15340 tablet_replica.cc:330] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: stopping tablet replica
I20260807 18:32:52.665882 15340 raft_consensus.cc:2229] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:32:52.666216 15342 tablet_replica.cc:330] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: stopping tablet replica
I20260807 18:32:52.666425 15340 raft_consensus.cc:2258] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:52.666724 15342 raft_consensus.cc:2229] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:32:52.667423 15342 raft_consensus.cc:2258] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:52.668874 15340 ts_tablet_manager.cc:1905] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:52.669523 15342 ts_tablet_manager.cc:1905] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:52.670174 15339 tablet_replica.cc:330] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: stopping tablet replica
I20260807 18:32:52.674445 15339 raft_consensus.cc:2229] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:32:52.675004 15339 raft_consensus.cc:2258] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:52.676350 15340 ts_tablet_manager.cc:1918] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:52.676507 15340 log.cc:1196] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/a948eefab9cf42d99b0ecff679ec80e9
I20260807 18:32:52.677078 15340 ts_tablet_manager.cc:1939] T a948eefab9cf42d99b0ecff679ec80e9 P bc8b72b593aa4eb5a3564c45c1c72237: Deleting consensus metadata
I20260807 18:32:52.678543 15342 ts_tablet_manager.cc:1918] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 07 18:32:52 dist-test-slave-1xrh krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:32:52.678817 15342 log.cc:1196] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/a948eefab9cf42d99b0ecff679ec80e9
I20260807 18:32:52.678872 14799 catalog_manager.cc:4841] TS bc8b72b593aa4eb5a3564c45c1c72237 (127.27.1.195:44969): tablet a948eefab9cf42d99b0ecff679ec80e9 (table test-table [id=284662a0cc4c41a2b9e0e1d7a633c697]) successfully deleted
I20260807 18:32:52.679669 15342 ts_tablet_manager.cc:1939] T a948eefab9cf42d99b0ecff679ec80e9 P 70e59774f3db4472aea8d5754951a4ed: Deleting consensus metadata
I20260807 18:32:52.681313 15339 ts_tablet_manager.cc:1905] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:52.681854 14797 catalog_manager.cc:4841] TS 70e59774f3db4472aea8d5754951a4ed (127.27.1.193:42325): tablet a948eefab9cf42d99b0ecff679ec80e9 (table test-table [id=284662a0cc4c41a2b9e0e1d7a633c697]) successfully deleted
I20260807 18:32:52.686966 15339 ts_tablet_manager.cc:1918] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:52.687176 15339 log.cc:1196] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/a948eefab9cf42d99b0ecff679ec80e9
I20260807 18:32:52.687738 15339 ts_tablet_manager.cc:1939] T a948eefab9cf42d99b0ecff679ec80e9 P e6f94ad6633846f2a564486fb3fd1c68: Deleting consensus metadata
I20260807 18:32:52.688257 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.653897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.27.1.193:42325)
0807 18:32:52.654779 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.654788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.665865 (+ 11077us) server_negotiation.cc:407] Connection header received
0807 18:32:52.666303 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.666312 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.666418 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.666510 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:52.668904 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.670272 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.672365 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.673143 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.673251 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.680231 (+ 6980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:52.680247 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:52.680266 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:52.680312 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:52.684635 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:52.685560 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:52.685568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:52.685573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:52.685768 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:52.686243 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:52.686249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:52.686253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:52.686722 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:52.687054 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.687558 (+ 504us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.687873 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701}
I20260807 18:32:52.689009 15008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:52.689550 14799 catalog_manager.cc:4841] TS e6f94ad6633846f2a564486fb3fd1c68 (127.27.1.194:46173): tablet a948eefab9cf42d99b0ecff679ec80e9 (table test-table [id=284662a0cc4c41a2b9e0e1d7a633c697]) successfully deleted
I20260807 18:32:52.786855 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 14879
I20260807 18:32:52.809520 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 15019
Aug 07 18:32:52 dist-test-slave-1xrh krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:32:52.824347 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.810348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.27.1.194:46173)
0807 18:32:52.810532 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.810537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.810555 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:32:52.810693 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.810697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.810761 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.810842 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:52.812188 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.812886 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.814091 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.814468 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.814565 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.818962 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:52.818977 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:52.818996 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:52.819032 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:52.821716 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:52.822348 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:52.822356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:52.822358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:52.822510 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:52.822824 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:52.822827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:52.822829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:52.823281 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:52.823510 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.823806 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.823971 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 18:32:52.825050 15148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:52.915925 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 15019
I20260807 18:32:52.936571 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 15159
Aug 07 18:32:52 dist-test-slave-1xrh krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:32:52.952070 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:52.937456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.27.1.195:44969)
0807 18:32:52.937708 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:52.937718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:52.937770 (+ 52us) server_negotiation.cc:407] Connection header received
0807 18:32:52.937950 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:52.937954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:52.938112 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:52.938226 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:52.939625 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.940335 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.941527 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:52.941890 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:52.941979 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:52.946515 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:52.946530 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:52.946547 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:52.946579 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:52.949505 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:52.950169 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:52.950177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:52.950179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:52.950315 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:52.950653 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:52.950656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:52.950658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:52.950981 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:52.951277 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:52.951525 (+ 248us) server_negotiation.cc:299] Negotiation successful
0807 18:32:52.951679 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 18:32:52.952739 15288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:53.042749 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 15159
I20260807 18:32:53.062750 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 14781
I20260807 18:32:53.063958 14847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:53.175390 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 14781
2026-08-07T18:32:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4799 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T18:32:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:32:53Z Disabled control of system clock
I20260807 18:32:53.267581 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38557
--webserver_interface=127.27.1.254
--webserver_port=0
--builtin_ntp_servers=127.27.1.212:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 18:32:53.608649 15351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:53.608938 15351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:53.609023 15351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:53.616217 15351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:32:53.616357 15351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:53.616430 15351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:53.616536 15351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:32:53.616617 15351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:32:53.625599 15351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:38557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:53.626942 15351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:53.628762 15351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:53.641088 15360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:53.641141 15356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:53.641153 15357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:53.641855 15351 server_base.cc:1034] running on GCE node
I20260807 18:32:53.642612 15351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:53.644068 15351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:53.645310 15351 hybrid_clock.cc:648] HybridClock initialized: now 1786127573645276 us; error 48 us; skew 500 ppm
I20260807 18:32:53.645664 15351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:32:53.647969 15351 webserver.cc:460] Webserver started at http://127.27.1.254:40701/ using document root <none> and password file <none>
I20260807 18:32:53.648653 15351 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:53.648792 15351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:53.649132 15351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:53.652069 15351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "e228b4b5be2b4f6ca3e4debca62c0723"
format_stamp: "Formatted at 2026-08-07 18:32:53 on dist-test-slave-1xrh"
server_key: "d7aab1f0d06a180483974db332518e55"
server_key_iv: "545c52d07119444a318bf5faf229cb70"
server_key_version: "encryptionkey@0"
I20260807 18:32:53.652891 15351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "e228b4b5be2b4f6ca3e4debca62c0723"
format_stamp: "Formatted at 2026-08-07 18:32:53 on dist-test-slave-1xrh"
server_key: "d7aab1f0d06a180483974db332518e55"
server_key_iv: "545c52d07119444a318bf5faf229cb70"
server_key_version: "encryptionkey@0"
I20260807 18:32:53.657989 15351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 18:32:53.661676 15366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:53.663048 15351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:32:53.663282 15351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "e228b4b5be2b4f6ca3e4debca62c0723"
format_stamp: "Formatted at 2026-08-07 18:32:53 on dist-test-slave-1xrh"
server_key: "d7aab1f0d06a180483974db332518e55"
server_key_iv: "545c52d07119444a318bf5faf229cb70"
server_key_version: "encryptionkey@0"
I20260807 18:32:53.663504 15351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:53.690690 15351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:53.691689 15351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:53.691996 15351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:53.711033 15351 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:38557
I20260807 18:32:53.711047 15420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:38557 every 8 connection(s)
I20260807 18:32:53.712450 15351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:32:53.716069 15422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:53.722738 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 15351
I20260807 18:32:53.722975 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:32:53.723449 27655 external_mini_cluster.cc:1428] Setting key fd809bdafa40322ea9bd6799187ba47f
I20260807 18:32:53.726740 15422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723: Bootstrap starting.
I20260807 18:32:53.730078 15422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:53.731154 15422 log.cc:824] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:53.734335 15422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723: No bootstrap required, opened a new log
I20260807 18:32:53.737424 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:53.725368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.27.1.254:38557)
0807 18:32:53.726347 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:53.726361 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:53.726431 (+ 70us) server_negotiation.cc:407] Connection header received
0807 18:32:53.728259 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:53.728280 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:53.728713 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:53.729015 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:53.729657 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:53.730997 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:53.732836 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:53.733511 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:53.733683 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:53.734989 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:53.734996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:53.735008 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:53.735640 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:53.735711 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:53.735734 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:53.735907 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:53.736154 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 18:32:53.736443 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":372,"threads_started":1}
I20260807 18:32:53.742724 15422 raft_consensus.cc:348] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e228b4b5be2b4f6ca3e4debca62c0723" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38557 } }
I20260807 18:32:53.743090 15422 raft_consensus.cc:374] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:53.743270 15422 raft_consensus.cc:729] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e228b4b5be2b4f6ca3e4debca62c0723, State: Initialized, Role: FOLLOWER
I20260807 18:32:53.743830 15422 consensus_queue.cc:260] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e228b4b5be2b4f6ca3e4debca62c0723" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38557 } }
I20260807 18:32:53.744084 15422 raft_consensus.cc:388] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:32:53.744258 15422 raft_consensus.cc:482] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:32:53.744436 15422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:53.746861 15422 raft_consensus.cc:504] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e228b4b5be2b4f6ca3e4debca62c0723" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38557 } }
I20260807 18:32:53.747391 15422 leader_election.cc:302] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e228b4b5be2b4f6ca3e4debca62c0723; no voters:
I20260807 18:32:53.747975 15422 leader_election.cc:288] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:32:53.748119 15427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:32:53.748822 15427 raft_consensus.cc:686] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [term 1 LEADER]: Becoming Leader. State: Replica: e228b4b5be2b4f6ca3e4debca62c0723, State: Running, Role: LEADER
I20260807 18:32:53.749384 15427 consensus_queue.cc:237] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e228b4b5be2b4f6ca3e4debca62c0723" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38557 } }
I20260807 18:32:53.750041 15422 sys_catalog.cc:564] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:32:53.752367 15428 sys_catalog.cc:455] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e228b4b5be2b4f6ca3e4debca62c0723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e228b4b5be2b4f6ca3e4debca62c0723" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38557 } } }
I20260807 18:32:53.752391 15429 sys_catalog.cc:455] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e228b4b5be2b4f6ca3e4debca62c0723. Latest consensus state: current_term: 1 leader_uuid: "e228b4b5be2b4f6ca3e4debca62c0723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e228b4b5be2b4f6ca3e4debca62c0723" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 38557 } } }
I20260807 18:32:53.752846 15428 sys_catalog.cc:458] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:53.752882 15429 sys_catalog.cc:458] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723 [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:53.755095 15432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:32:53.760284 15432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:32:53.769752 15432 catalog_manager.cc:1347] Generated new cluster ID: ee9e9998e5514767b6162ae905222134
I20260807 18:32:53.769887 15432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:32:53.797597 15432 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:32:53.799093 15432 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:32:53.809482 15432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723: Generated new TSK 0
I20260807 18:32:53.810356 15432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:32:53.828030 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38557
--builtin_ntp_servers=127.27.1.212:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 18:32:54.168886 15448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:54.169162 15448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:54.169328 15448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:54.176497 15448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:54.176599 15448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:54.176750 15448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:32:54.186277 15448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:54.187683 15448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:54.189520 15448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:54.207955 15454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:54.207978 15455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:54.207978 15457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:54.208842 15448 server_base.cc:1034] running on GCE node
I20260807 18:32:54.209456 15448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:54.210419 15448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:54.211654 15448 hybrid_clock.cc:648] HybridClock initialized: now 1786127574211616 us; error 54 us; skew 500 ppm
I20260807 18:32:54.212006 15448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:32:54.214268 15448 webserver.cc:460] Webserver started at http://127.27.1.193:38987/ using document root <none> and password file <none>
I20260807 18:32:54.214970 15448 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:54.215117 15448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:54.215504 15448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:54.218556 15448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "95fb534d2edf4220b53ef9ca7c76fb00"
format_stamp: "Formatted at 2026-08-07 18:32:54 on dist-test-slave-1xrh"
server_key: "2ca8a3c85ed673fa1ef79eeb7f120d09"
server_key_iv: "233ca91e7ff3852e6b01df2b10b51749"
server_key_version: "encryptionkey@0"
I20260807 18:32:54.219472 15448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "95fb534d2edf4220b53ef9ca7c76fb00"
format_stamp: "Formatted at 2026-08-07 18:32:54 on dist-test-slave-1xrh"
server_key: "2ca8a3c85ed673fa1ef79eeb7f120d09"
server_key_iv: "233ca91e7ff3852e6b01df2b10b51749"
server_key_version: "encryptionkey@0"
I20260807 18:32:54.224659 15448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:32:54.228097 15463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:54.229478 15448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:32:54.229667 15448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "95fb534d2edf4220b53ef9ca7c76fb00"
format_stamp: "Formatted at 2026-08-07 18:32:54 on dist-test-slave-1xrh"
server_key: "2ca8a3c85ed673fa1ef79eeb7f120d09"
server_key_iv: "233ca91e7ff3852e6b01df2b10b51749"
server_key_version: "encryptionkey@0"
I20260807 18:32:54.229892 15448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:54.242578 15448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:54.243458 15448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:54.243764 15448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:54.244731 15448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:54.246176 15448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:54.246304 15448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:54.246449 15448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:54.246541 15448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:54.282786 15448 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:44931
I20260807 18:32:54.282853 15583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:44931 every 8 connection(s)
I20260807 18:32:54.284186 15448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:32:54.290876 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 15448
I20260807 18:32:54.291057 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:32:54.291617 27655 external_mini_cluster.cc:1428] Setting key 068289e274fc59d034ddb4c155382723
I20260807 18:32:54.294261 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38557
--builtin_ntp_servers=127.27.1.212:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 18:32:54.297089 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:54.286725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:34047 (local address 127.27.1.254:38557)
0807 18:32:54.287327 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:54.287331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:54.288417 (+ 1086us) server_negotiation.cc:407] Connection header received
0807 18:32:54.290233 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:54.290239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:54.290333 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:54.290456 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:54.291632 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:54.292424 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:54.293980 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:54.294376 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:54.294523 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:54.296111 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:54.296120 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:54.296128 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:54.296153 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:54.296224 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:54.296243 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:54.296350 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:54.296559 (+ 209us) server_negotiation.cc:299] Negotiation successful
0807 18:32:54.296665 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":310,"threads_started":1}
I20260807 18:32:54.297906 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:54.287363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38557 (local address 127.27.1.193:34047)
0807 18:32:54.288249 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:54.288285 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:54.289953 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:54.290696 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:54.290705 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:54.290795 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:54.291455 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:54.291473 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:54.292589 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:54.292599 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:54.293831 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:54.293844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:54.294756 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:54.294761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:54.294972 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:54.295738 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:32:54.295741 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:54.295784 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:32:54.295917 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:54.296351 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:54.296359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:54.296365 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 18:32:54.296499 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 18:32:54.296767 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":288,"threads_started":1}
I20260807 18:32:54.299577 15584 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38557
I20260807 18:32:54.299898 15584 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:54.300671 15584 heartbeater.cc:511] Master 127.27.1.254:38557 requested a full tablet report, sending...
I20260807 18:32:54.302812 15383 ts_manager.cc:194] Registered new tserver with Master: 95fb534d2edf4220b53ef9ca7c76fb00 (127.27.1.193:44931)
I20260807 18:32:54.304375 15383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.193:34047
W20260807 18:32:54.639882 15588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:54.640159 15588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:54.640333 15588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:54.647600 15588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:54.647689 15588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:54.647882 15588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:32:54.657136 15588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:54.658447 15588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:54.660282 15588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:54.673846 15594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:54.673801 15597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:54.673756 15595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:54.674604 15588 server_base.cc:1034] running on GCE node
I20260807 18:32:54.675256 15588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:54.676157 15588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:54.677357 15588 hybrid_clock.cc:648] HybridClock initialized: now 1786127574677304 us; error 45 us; skew 500 ppm
I20260807 18:32:54.677665 15588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:32:54.679852 15588 webserver.cc:460] Webserver started at http://127.27.1.194:36057/ using document root <none> and password file <none>
I20260807 18:32:54.680538 15588 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:54.680662 15588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:54.680979 15588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:54.683857 15588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "560ac052bbdc402b9e4f8d5ef55381f9"
format_stamp: "Formatted at 2026-08-07 18:32:54 on dist-test-slave-1xrh"
server_key: "d17536b8fc9539e36b2ebe2dea92fe16"
server_key_iv: "d26a356f44414acbb40d7d2b15044c5c"
server_key_version: "encryptionkey@0"
I20260807 18:32:54.684667 15588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "560ac052bbdc402b9e4f8d5ef55381f9"
format_stamp: "Formatted at 2026-08-07 18:32:54 on dist-test-slave-1xrh"
server_key: "d17536b8fc9539e36b2ebe2dea92fe16"
server_key_iv: "d26a356f44414acbb40d7d2b15044c5c"
server_key_version: "encryptionkey@0"
I20260807 18:32:54.689651 15588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:32:54.693074 15604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:54.694347 15588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:32:54.694515 15588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "560ac052bbdc402b9e4f8d5ef55381f9"
format_stamp: "Formatted at 2026-08-07 18:32:54 on dist-test-slave-1xrh"
server_key: "d17536b8fc9539e36b2ebe2dea92fe16"
server_key_iv: "d26a356f44414acbb40d7d2b15044c5c"
server_key_version: "encryptionkey@0"
I20260807 18:32:54.694718 15588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:54.718026 15588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:54.718792 15588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:54.719098 15588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:54.720100 15588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:54.721511 15588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:54.721596 15588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:54.721766 15588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:54.721866 15588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:54.755400 15588 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:34111
I20260807 18:32:54.755505 15718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:34111 every 8 connection(s)
I20260807 18:32:54.756726 15588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:32:54.766665 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 15588
I20260807 18:32:54.766852 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:32:54.767444 27655 external_mini_cluster.cc:1428] Setting key fb5f1c92d6bf13c941049407c0b8d43c
I20260807 18:32:54.769802 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:54.759370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:55081 (local address 127.27.1.254:38557)
0807 18:32:54.759568 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:54.759573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:54.760880 (+ 1307us) server_negotiation.cc:407] Connection header received
0807 18:32:54.762695 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:54.762701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:54.762794 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:54.762913 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:54.764048 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:54.764798 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:54.766586 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:54.767111 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:54.767269 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:54.768988 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:54.768994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:54.768998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:54.769018 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:54.769056 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:54.769071 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:54.769168 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:54.769459 (+ 291us) server_negotiation.cc:299] Negotiation successful
0807 18:32:54.769562 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 18:32:54.770676 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38557
--builtin_ntp_servers=127.27.1.212:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 18:32:54.771257 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:54.759826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38557 (local address 127.27.1.194:55081)
0807 18:32:54.760708 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:54.760750 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:54.762410 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:54.763203 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:54.763211 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:54.763302 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:54.763863 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:54.763879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:54.764962 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:54.764968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:54.766261 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:54.766278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:54.767385 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:54.767393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:54.767739 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:54.768673 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:32:54.768676 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:54.768722 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:32:54.768800 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:54.769204 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:54.769214 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:54.769223 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 18:32:54.769408 (+ 185us) client_negotiation.cc:240] Negotiation successful
0807 18:32:54.769828 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":286,"threads_started":1}
I20260807 18:32:54.772619 15719 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38557
I20260807 18:32:54.772944 15719 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:54.773800 15719 heartbeater.cc:511] Master 127.27.1.254:38557 requested a full tablet report, sending...
I20260807 18:32:54.775336 15383 ts_manager.cc:194] Registered new tserver with Master: 560ac052bbdc402b9e4f8d5ef55381f9 (127.27.1.194:34111)
I20260807 18:32:54.776115 15383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.194:55081
W20260807 18:32:55.105983 15722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:55.106259 15722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:55.106424 15722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:55.113605 15722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:55.113698 15722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:55.113891 15722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:32:55.122879 15722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:38557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:55.124171 15722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:55.125941 15722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:55.138655 15727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:55.138717 15731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:55.142204 15722 server_base.cc:1034] running on GCE node
W20260807 18:32:55.138788 15728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:55.143075 15722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:55.144045 15722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:55.145249 15722 hybrid_clock.cc:648] HybridClock initialized: now 1786127575145206 us; error 30 us; skew 500 ppm
I20260807 18:32:55.145584 15722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:32:55.148092 15722 webserver.cc:460] Webserver started at http://127.27.1.195:46699/ using document root <none> and password file <none>
I20260807 18:32:55.148788 15722 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:55.148918 15722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:55.149258 15722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:55.152402 15722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "50b41417abff48b1875e13e2898d99ba"
format_stamp: "Formatted at 2026-08-07 18:32:55 on dist-test-slave-1xrh"
server_key: "6e4635873dc05368644ff576887c949b"
server_key_iv: "a9e61e368fefade70651ba20c42a52f1"
server_key_version: "encryptionkey@0"
I20260807 18:32:55.153182 15722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "50b41417abff48b1875e13e2898d99ba"
format_stamp: "Formatted at 2026-08-07 18:32:55 on dist-test-slave-1xrh"
server_key: "6e4635873dc05368644ff576887c949b"
server_key_iv: "a9e61e368fefade70651ba20c42a52f1"
server_key_version: "encryptionkey@0"
I20260807 18:32:55.158878 15722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 18:32:55.162545 15737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:55.163908 15722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260807 18:32:55.164105 15722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "50b41417abff48b1875e13e2898d99ba"
format_stamp: "Formatted at 2026-08-07 18:32:55 on dist-test-slave-1xrh"
server_key: "6e4635873dc05368644ff576887c949b"
server_key_iv: "a9e61e368fefade70651ba20c42a52f1"
server_key_version: "encryptionkey@0"
I20260807 18:32:55.164321 15722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:55.184231 15722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:55.185070 15722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:55.185391 15722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:55.186398 15722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:55.187952 15722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:55.188072 15722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:55.188211 15722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:55.188306 15722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:55.222926 15722 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:38017
I20260807 18:32:55.222946 15853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:38017 every 8 connection(s)
I20260807 18:32:55.224293 15722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:32:55.231549 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 15722
I20260807 18:32:55.231729 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:32:55.232196 27655 external_mini_cluster.cc:1428] Setting key 446c1fad17ea79424e65df5ca256beb1
I20260807 18:32:55.236847 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.226827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:60095 (local address 127.27.1.254:38557)
0807 18:32:55.227013 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.227017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.228320 (+ 1303us) server_negotiation.cc:407] Connection header received
0807 18:32:55.229978 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.229983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.230059 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.230148 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:55.231199 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.232331 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.234027 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.234490 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.234641 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.236021 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:55.236027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:55.236031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:55.236056 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:55.236085 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:55.236097 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:55.236197 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.236420 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.236517 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 18:32:55.237743 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.227331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:38557 (local address 127.27.1.195:60095)
0807 18:32:55.228183 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:55.228219 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:55.229743 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:55.230335 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:55.230342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:55.230418 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:55.230986 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.231010 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.232485 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.232491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.233872 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.233890 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.234729 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.234738 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.235044 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.235715 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:32:55.235718 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:55.235754 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:32:55.235828 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:55.236198 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:55.236211 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:55.236221 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 18:32:55.236408 (+ 187us) client_negotiation.cc:240] Negotiation successful
0807 18:32:55.236765 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":274,"threads_started":1}
I20260807 18:32:55.238965 15854 heartbeater.cc:347] Connected to a master server at 127.27.1.254:38557
I20260807 18:32:55.239300 15854 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:55.239969 15854 heartbeater.cc:511] Master 127.27.1.254:38557 requested a full tablet report, sending...
I20260807 18:32:55.241394 15383 ts_manager.cc:194] Registered new tserver with Master: 50b41417abff48b1875e13e2898d99ba (127.27.1.195:38017)
I20260807 18:32:55.242228 15383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.195:60095
I20260807 18:32:55.246940 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:32:55.256582 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.250985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.27.1.254:38557)
0807 18:32:55.251233 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.251237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.251667 (+ 430us) server_negotiation.cc:407] Connection header received
0807 18:32:55.251794 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.251797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.251866 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.251953 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:55.252379 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.253063 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.254403 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.254786 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.254877 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.255912 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:55.255917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:55.255921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:55.255937 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:55.255968 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:55.255979 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:55.256096 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.256326 (+ 230us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.256370 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:32:55.260444 15383 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:32:55.267112 15383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:32:55.306203 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.279463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.27.1.194:34111)
0807 18:32:55.290210 (+ 10747us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.290222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.295370 (+ 5148us) server_negotiation.cc:407] Connection header received
0807 18:32:55.295874 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.295884 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.296161 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.296376 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:55.297715 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.299710 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.303403 (+ 3693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.304937 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.305074 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.305355 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.305661 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.305852 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10561,"thread_start_us":10439,"threads_started":1}
I20260807 18:32:55.306382 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.279450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:34111 (local address 127.0.0.1:54404)
0807 18:32:55.295211 (+ 15761us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:55.295236 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:55.295654 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:55.296756 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:55.296767 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:55.296809 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:55.297439 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.297456 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.299954 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.299962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.303194 (+ 3232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.303215 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.305145 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.305152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.305398 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.305455 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 18:32:55.305600 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 18:32:55.305910 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15557,"mutex_wait_us":105,"thread_start_us":15963,"threads_started":1}
I20260807 18:32:55.307073 15584 heartbeater.cc:503] Master 127.27.1.254:38557 was elected leader, sending a full tablet report...
I20260807 18:32:55.310413 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.279061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:44931 (local address 127.0.0.1:47066)
0807 18:32:55.290678 (+ 11617us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:55.290727 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:55.290936 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:55.302856 (+ 11920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:55.302860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:55.302885 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:55.303342 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.303355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.305290 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.305295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.307524 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.307541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.309709 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.309715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.309875 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.309906 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 18:32:55.309986 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 18:32:55.310121 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11294,"thread_start_us":12063,"threads_started":1}
I20260807 18:32:55.310485 15653 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9fec91e77e468abc9b488a20bbd297 (DEFAULT_TABLE table=test-table [id=9ffe9cae3b78491d96c12ab86f2ed8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:55.311195 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.279297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47066 (local address 127.27.1.193:44931)
0807 18:32:55.302033 (+ 22736us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.302042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.302072 (+ 30us) server_negotiation.cc:407] Connection header received
0807 18:32:55.302168 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.302177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.302396 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.302605 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:55.303576 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.305126 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.307783 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.309193 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.309322 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.309575 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.310668 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.310832 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22551,"thread_start_us":23433,"threads_started":1}
I20260807 18:32:55.312170 15653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d9fec91e77e468abc9b488a20bbd297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:55.314787 15517 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9fec91e77e468abc9b488a20bbd297 (DEFAULT_TABLE table=test-table [id=9ffe9cae3b78491d96c12ab86f2ed8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:55.316172 15517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d9fec91e77e468abc9b488a20bbd297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:55.317260 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.292008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:38017 (local address 127.0.0.1:55878)
0807 18:32:55.301165 (+ 9157us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:55.301188 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:55.301339 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:55.304061 (+ 2722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:55.304065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:55.304092 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:55.304464 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.304474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.306247 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.306252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.314426 (+ 8174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.314442 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.316569 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.316576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.316753 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.316786 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 18:32:55.316867 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 18:32:55.316991 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8994,"thread_start_us":9426,"threads_started":1}
I20260807 18:32:55.318022 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.292428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.27.1.195:38017)
0807 18:32:55.303420 (+ 10992us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.303439 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.303475 (+ 36us) server_negotiation.cc:407] Connection header received
0807 18:32:55.303689 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.303695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.303869 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.304065 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:55.304646 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.306075 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.314659 (+ 8584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.316068 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.316201 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.316452 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.317491 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.317660 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10815,"thread_start_us":10625,"threads_started":1}
I20260807 18:32:55.323966 15787 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9fec91e77e468abc9b488a20bbd297 (DEFAULT_TABLE table=test-table [id=9ffe9cae3b78491d96c12ab86f2ed8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:55.325122 15787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d9fec91e77e468abc9b488a20bbd297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:55.328421 15872 tablet_bootstrap.cc:492] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: Bootstrap starting.
I20260807 18:32:55.330569 15873 tablet_bootstrap.cc:492] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: Bootstrap starting.
I20260807 18:32:55.332643 15872 tablet_bootstrap.cc:654] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:55.333679 15873 tablet_bootstrap.cc:654] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:55.334504 15872 log.cc:824] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:55.334853 15873 log.cc:824] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:55.337801 15872 tablet_bootstrap.cc:492] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: No bootstrap required, opened a new log
I20260807 18:32:55.337797 15873 tablet_bootstrap.cc:492] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: No bootstrap required, opened a new log
I20260807 18:32:55.338076 15872 ts_tablet_manager.cc:1397] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260807 18:32:55.338133 15873 ts_tablet_manager.cc:1397] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 18:32:55.339639 15876 tablet_bootstrap.cc:492] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: Bootstrap starting.
I20260807 18:32:55.342646 15876 tablet_bootstrap.cc:654] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:55.343804 15876 log.cc:824] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:55.349148 15872 raft_consensus.cc:348] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44931 } } peers { permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 } } peers { permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 } }
I20260807 18:32:55.349152 15873 raft_consensus.cc:348] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44931 } } peers { permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 } } peers { permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 } }
I20260807 18:32:55.349603 15872 raft_consensus.cc:374] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:55.349632 15876 tablet_bootstrap.cc:492] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: No bootstrap required, opened a new log
I20260807 18:32:55.349642 15873 raft_consensus.cc:374] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:55.349715 15872 raft_consensus.cc:729] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560ac052bbdc402b9e4f8d5ef55381f9, State: Initialized, Role: FOLLOWER
I20260807 18:32:55.349730 15873 raft_consensus.cc:729] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fb534d2edf4220b53ef9ca7c76fb00, State: Initialized, Role: FOLLOWER
I20260807 18:32:55.349926 15876 ts_tablet_manager.cc:1397] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260807 18:32:55.350419 15873 consensus_queue.cc:260] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44931 } } peers { permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 } } peers { permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 } }
I20260807 18:32:55.350412 15872 consensus_queue.cc:260] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44931 } } peers { permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 } } peers { permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 } }
I20260807 18:32:55.352514 15873 ts_tablet_manager.cc:1428] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260807 18:32:55.359477 15876 raft_consensus.cc:348] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44931 } } peers { permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 } } peers { permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 } }
I20260807 18:32:55.359941 15876 raft_consensus.cc:374] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:55.360049 15876 raft_consensus.cc:729] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b41417abff48b1875e13e2898d99ba, State: Initialized, Role: FOLLOWER
I20260807 18:32:55.360388 15719 heartbeater.cc:503] Master 127.27.1.254:38557 was elected leader, sending a full tablet report...
I20260807 18:32:55.360889 15872 ts_tablet_manager.cc:1428] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: Time spent starting tablet: real 0.023s user 0.005s sys 0.018s
I20260807 18:32:55.360800 15876 consensus_queue.cc:260] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44931 } } peers { permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 } } peers { permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 } }
I20260807 18:32:55.361833 15854 heartbeater.cc:503] Master 127.27.1.254:38557 was elected leader, sending a full tablet report...
I20260807 18:32:55.363554 15876 ts_tablet_manager.cc:1428] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260807 18:32:55.469942 15879 raft_consensus.cc:482] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:32:55.470371 15879 raft_consensus.cc:504] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44931 } } peers { permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 } } peers { permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 } }
I20260807 18:32:55.472566 15879 leader_election.cc:288] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 560ac052bbdc402b9e4f8d5ef55381f9 (127.27.1.194:34111), 50b41417abff48b1875e13e2898d99ba (127.27.1.195:38017)
W20260807 18:32:55.476009 15855 tablet.cc:2369] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:32:55.480767 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.473007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:38017 (local address 127.27.1.193:55467)
0807 18:32:55.473958 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:55.473983 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:55.474174 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:55.475421 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:55.475428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:55.475457 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:55.475877 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.475905 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.477368 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.477373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.478788 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.478800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.480021 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.480027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.480189 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.480212 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 18:32:55.480331 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 18:32:55.480458 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":1759,"threads_started":1}
I20260807 18:32:55.480767 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.474514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:55467 (local address 127.27.1.195:38017)
0807 18:32:55.474806 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.474813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.474862 (+ 49us) server_negotiation.cc:407] Connection header received
0807 18:32:55.474977 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.474982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.475072 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.475219 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:55.476090 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.477145 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.478988 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.479812 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.479994 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.480067 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.480356 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.480482 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 18:32:55.481906 15807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9fec91e77e468abc9b488a20bbd297" candidate_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b41417abff48b1875e13e2898d99ba" is_pre_election: true
I20260807 18:32:55.482402 15807 raft_consensus.cc:2454] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95fb534d2edf4220b53ef9ca7c76fb00 in term 0.
I20260807 18:32:55.483088 15465 leader_election.cc:302] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b41417abff48b1875e13e2898d99ba, 95fb534d2edf4220b53ef9ca7c76fb00; no voters:
I20260807 18:32:55.483426 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.473113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:51251 (local address 127.27.1.194:34111)
0807 18:32:55.473354 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.473361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.475187 (+ 1826us) server_negotiation.cc:407] Connection header received
0807 18:32:55.475406 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.475412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.475507 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.475608 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:55.476379 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.477697 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.480663 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.482110 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.482257 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.482345 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.483011 (+ 666us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.483182 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 18:32:55.483435 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.473093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:34111 (local address 127.27.1.193:51251)
0807 18:32:55.475069 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:55.475084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:55.475254 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:55.475658 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:55.475664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:55.475700 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:55.476180 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.476197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.477944 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.477951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.480377 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:55.480395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.482547 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.482557 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:55.482769 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.482814 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 18:32:55.482959 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 18:32:55.483095 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1858,"thread_start_us":437,"threads_started":1}
I20260807 18:32:55.483531 15879 raft_consensus.cc:2781] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:32:55.483676 15879 raft_consensus.cc:482] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:32:55.483800 15879 raft_consensus.cc:3037] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:55.484316 15673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9fec91e77e468abc9b488a20bbd297" candidate_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" is_pre_election: true
I20260807 18:32:55.484776 15673 raft_consensus.cc:2454] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95fb534d2edf4220b53ef9ca7c76fb00 in term 0.
I20260807 18:32:55.486166 15879 raft_consensus.cc:504] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44931 } } peers { permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 } } peers { permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 } }
I20260807 18:32:55.486709 15879 leader_election.cc:288] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [CANDIDATE]: Term 1 election: Requested vote from peers 560ac052bbdc402b9e4f8d5ef55381f9 (127.27.1.194:34111), 50b41417abff48b1875e13e2898d99ba (127.27.1.195:38017)
I20260807 18:32:55.487182 15673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9fec91e77e468abc9b488a20bbd297" candidate_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560ac052bbdc402b9e4f8d5ef55381f9"
I20260807 18:32:55.487377 15673 raft_consensus.cc:3037] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:55.487677 15807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9fec91e77e468abc9b488a20bbd297" candidate_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b41417abff48b1875e13e2898d99ba"
I20260807 18:32:55.487857 15807 raft_consensus.cc:3037] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:55.489678 15673 raft_consensus.cc:2454] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fb534d2edf4220b53ef9ca7c76fb00 in term 1.
I20260807 18:32:55.490057 15807 raft_consensus.cc:2454] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fb534d2edf4220b53ef9ca7c76fb00 in term 1.
I20260807 18:32:55.490087 15467 leader_election.cc:302] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 560ac052bbdc402b9e4f8d5ef55381f9, 95fb534d2edf4220b53ef9ca7c76fb00; no voters:
I20260807 18:32:55.490363 15879 raft_consensus.cc:2781] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:32:55.490803 15879 raft_consensus.cc:686] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 1 LEADER]: Becoming Leader. State: Replica: 95fb534d2edf4220b53ef9ca7c76fb00, State: Running, Role: LEADER
I20260807 18:32:55.491322 15879 consensus_queue.cc:237] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44931 } } peers { permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 } } peers { permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 } }
I20260807 18:32:55.496055 15383 catalog_manager.cc:5495] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95fb534d2edf4220b53ef9ca7c76fb00 (127.27.1.193). New cstate: current_term: 1 leader_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fb534d2edf4220b53ef9ca7c76fb00" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 44931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 } health_report { overall_health: UNKNOWN } } }
W20260807 18:32:55.508814 15720 tablet.cc:2369] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:32:55.530176 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.524459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.27.1.193:44931)
0807 18:32:55.524658 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.524664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.524711 (+ 47us) server_negotiation.cc:407] Connection header received
0807 18:32:55.524968 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.524972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.525091 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.525248 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:32:55.525637 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.526354 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.527876 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.528296 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.528441 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.529101 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:32:55.529579 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:32:55.529783 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.529903 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.529947 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260807 18:32:55.536653 15585 tablet.cc:2369] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:32:55.538275 15673 raft_consensus.cc:1261] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9 [term 1 FOLLOWER]: Refusing update from remote peer 95fb534d2edf4220b53ef9ca7c76fb00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:32:55.538277 15807 raft_consensus.cc:1261] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba [term 1 FOLLOWER]: Refusing update from remote peer 95fb534d2edf4220b53ef9ca7c76fb00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:32:55.539212 15883 consensus_queue.cc:1035] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "560ac052bbdc402b9e4f8d5ef55381f9" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:32:55.539554 15879 consensus_queue.cc:1035] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b41417abff48b1875e13e2898d99ba" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:32:55.550700 15886 mvcc.cc:205] Tried to move back new op lower bound from 7315978549395243008 to 7315978549216239616. Current Snapshot: MvccSnapshot[applied={T|T < 7315978549395243008}]
I20260807 18:32:55.566593 15382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:32:55.566942 15382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:32:55.571005 15382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e228b4b5be2b4f6ca3e4debca62c0723: Sending DeleteTablet for 3 replicas of tablet 7d9fec91e77e468abc9b488a20bbd297
I20260807 18:32:55.572134 15517 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d9fec91e77e468abc9b488a20bbd297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:55 UTC) from {username='slave'} at 127.0.0.1:47066
I20260807 18:32:55.572625 15653 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d9fec91e77e468abc9b488a20bbd297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:55 UTC) from {username='slave'} at 127.0.0.1:54404
I20260807 18:32:55.573079 15895 tablet_replica.cc:330] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: stopping tablet replica
I20260807 18:32:55.573539 15895 raft_consensus.cc:2229] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:32:55.573616 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 15448
I20260807 18:32:55.573731 15787 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d9fec91e77e468abc9b488a20bbd297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:32:55 UTC) from {username='slave'} at 127.0.0.1:55878
I20260807 18:32:55.574299 15895 raft_consensus.cc:2258] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:55.576498 15895 ts_tablet_manager.cc:1905] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:55.586115 15895 ts_tablet_manager.cc:1918] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:55.586311 15895 log.cc:1196] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/7d9fec91e77e468abc9b488a20bbd297
I20260807 18:32:55.586969 15895 ts_tablet_manager.cc:1939] T 7d9fec91e77e468abc9b488a20bbd297 P 95fb534d2edf4220b53ef9ca7c76fb00: Deleting consensus metadata
I20260807 18:32:55.588912 15368 catalog_manager.cc:4841] TS 95fb534d2edf4220b53ef9ca7c76fb00 (127.27.1.193:44931): tablet 7d9fec91e77e468abc9b488a20bbd297 (table test-table [id=9ffe9cae3b78491d96c12ab86f2ed8d3]) successfully deleted
I20260807 18:32:55.593536 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.575817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.27.1.193:44931)
0807 18:32:55.575994 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.576000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.585758 (+ 9758us) server_negotiation.cc:407] Connection header received
0807 18:32:55.586250 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.586255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.586366 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.586453 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:55.587223 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.588486 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.590425 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.590846 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.590996 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.592140 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:55.592149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:55.592162 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:55.592722 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:55.592768 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:55.592787 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:55.592930 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.593066 (+ 136us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.593161 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 18:32:55.594235 15579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:55.595928 15897 tablet_replica.cc:330] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: stopping tablet replica
I20260807 18:32:55.596525 15897 raft_consensus.cc:2229] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:32:55.597914 15896 tablet_replica.cc:330] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: stopping tablet replica
I20260807 18:32:55.598351 15896 raft_consensus.cc:2229] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:32:55.598817 15896 raft_consensus.cc:2258] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:55.597256 15897 raft_consensus.cc:2258] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:32:55.600942 15897 ts_tablet_manager.cc:1905] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:55.601116 15896 ts_tablet_manager.cc:1905] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:32:55.605829 15897 ts_tablet_manager.cc:1918] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:55.606007 15897 log.cc:1196] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/7d9fec91e77e468abc9b488a20bbd297
I20260807 18:32:55.606498 15897 ts_tablet_manager.cc:1939] T 7d9fec91e77e468abc9b488a20bbd297 P 50b41417abff48b1875e13e2898d99ba: Deleting consensus metadata
I20260807 18:32:55.608176 15368 catalog_manager.cc:4841] TS 50b41417abff48b1875e13e2898d99ba (127.27.1.195:38017): tablet 7d9fec91e77e468abc9b488a20bbd297 (table test-table [id=9ffe9cae3b78491d96c12ab86f2ed8d3]) successfully deleted
I20260807 18:32:55.608507 15896 ts_tablet_manager.cc:1918] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:32:55.608685 15896 log.cc:1196] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/7d9fec91e77e468abc9b488a20bbd297
I20260807 18:32:55.609241 15896 ts_tablet_manager.cc:1939] T 7d9fec91e77e468abc9b488a20bbd297 P 560ac052bbdc402b9e4f8d5ef55381f9: Deleting consensus metadata
I20260807 18:32:55.611064 15369 catalog_manager.cc:4841] TS 560ac052bbdc402b9e4f8d5ef55381f9 (127.27.1.194:34111): tablet 7d9fec91e77e468abc9b488a20bbd297 (table test-table [id=9ffe9cae3b78491d96c12ab86f2ed8d3]) successfully deleted
I20260807 18:32:55.694311 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 15448
I20260807 18:32:55.717558 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 15588
I20260807 18:32:55.725692 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.718342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.27.1.194:34111)
0807 18:32:55.718525 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.718530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.718612 (+ 82us) server_negotiation.cc:407] Connection header received
0807 18:32:55.718801 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.718805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.718936 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.719018 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:55.719537 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.720727 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.722659 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.723066 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.723235 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.724337 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:55.724346 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:55.724361 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:55.724946 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:55.725010 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:55.725030 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:55.725116 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.725273 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.725364 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 18:32:55.726399 15714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:55.815066 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 15588
I20260807 18:32:55.838646 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 15722
I20260807 18:32:55.846365 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:55.839495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.27.1.195:38017)
0807 18:32:55.839714 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:55.839719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:55.839742 (+ 23us) server_negotiation.cc:407] Connection header received
0807 18:32:55.839909 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:55.839916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:55.840027 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:55.840143 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:55.840597 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.841545 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.843068 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:55.843669 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:55.843858 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:55.844996 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:55.845006 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:55.845023 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:32:55.845598 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:55.845649 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:32:55.845669 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:55.845809 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:55.845939 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 18:32:55.846066 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:32:55.847116 15849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:55.936496 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 15722
I20260807 18:32:55.958207 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 15351
I20260807 18:32:55.959400 15416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:32:56.073437 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 15351
2026-08-07T18:32:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2899 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:32:56 dist-test-slave-1xrh krb5kdc[15906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:32:56 dist-test-slave-1xrh krb5kdc[15906](info): set up 2 sockets
Aug 07 18:32:56 dist-test-slave-1xrh krb5kdc[15906](info): commencing operation
krb5kdc: starting...
W20260807 18:32:57.260958 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.002s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:32:57 dist-test-slave-1xrh krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:32:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:32:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:32:57.566768 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34125
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:34125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:57.905951 15920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:57.906316 15920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:57.906493 15920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:57.913839 15920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:32:57.913995 15920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:57.914108 15920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:57.914227 15920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:32:57.914314 15920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:32:57.923985 15920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:34125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:57.925385 15920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:57.926697 15920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:57.938897 15926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:57.938926 15929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:57.938936 15927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:57.939531 15920 server_base.cc:1034] running on GCE node
I20260807 18:32:57.940418 15920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:57.941783 15920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:57.943034 15920 hybrid_clock.cc:648] HybridClock initialized: now 1786127577942981 us; error 58 us; skew 500 ppm
I20260807 18:32:57.943394 15920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:57 dist-test-slave-1xrh krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:57.946620 15920 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:32:57.948216 15920 webserver.cc:460] Webserver started at http://127.27.1.254:35947/ using document root <none> and password file <none>
I20260807 18:32:57.948889 15920 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:57.949038 15920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:57.949345 15920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:57.952345 15920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "713a109be29644d2b71b7c490979f109"
format_stamp: "Formatted at 2026-08-07 18:32:57 on dist-test-slave-1xrh"
server_key: "bc650627e68a5b351c87f9018ac90193"
server_key_iv: "c5645af6199eda6a485cf857d671cbe0"
server_key_version: "encryptionkey@0"
I20260807 18:32:57.953107 15920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "713a109be29644d2b71b7c490979f109"
format_stamp: "Formatted at 2026-08-07 18:32:57 on dist-test-slave-1xrh"
server_key: "bc650627e68a5b351c87f9018ac90193"
server_key_iv: "c5645af6199eda6a485cf857d671cbe0"
server_key_version: "encryptionkey@0"
I20260807 18:32:57.958201 15920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:32:57.961591 15936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:57.962831 15920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:32:57.963019 15920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "713a109be29644d2b71b7c490979f109"
format_stamp: "Formatted at 2026-08-07 18:32:57 on dist-test-slave-1xrh"
server_key: "bc650627e68a5b351c87f9018ac90193"
server_key_iv: "c5645af6199eda6a485cf857d671cbe0"
server_key_version: "encryptionkey@0"
I20260807 18:32:57.963297 15920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:57.979318 15920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:57.983767 15920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:57.984081 15920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:58.002807 15920 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:34125
I20260807 18:32:58.002846 15988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:34125 every 8 connection(s)
I20260807 18:32:58.004328 15920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:32:58.006824 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 15920
I20260807 18:32:58.007019 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:32:58.007529 27655 external_mini_cluster.cc:1428] Setting key 964f2c0dcca0711f36add32ba0e32bb9
I20260807 18:32:58.008375 15989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 18:32:58 dist-test-slave-1xrh krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:58.023523 15989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109: Bootstrap starting.
I20260807 18:32:58.027899 15989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:58.029189 15989 log.cc:824] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:58.032262 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:58.009624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.27.1.254:34125)
0807 18:32:58.010381 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:58.010393 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:58.010433 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:32:58.012078 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:58.012099 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:58.012477 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:58.012797 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:58.014694 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:58.016049 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:58.017801 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:58.018433 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:58.018568 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:58.023543 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:58.023557 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:58.023576 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:58.023621 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:58.027660 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:58.028397 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:58.028405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:58.028413 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:58.028620 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:58.028977 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:58.028982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:58.028987 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:58.029721 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:58.030047 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:58.030497 (+ 450us) server_negotiation.cc:299] Negotiation successful
0807 18:32:58.030951 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":346,"threads_started":1}
I20260807 18:32:58.032759 15989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109: No bootstrap required, opened a new log
I20260807 18:32:58.040889 15989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713a109be29644d2b71b7c490979f109" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34125 } }
I20260807 18:32:58.041237 15989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:58.041311 15989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713a109be29644d2b71b7c490979f109, State: Initialized, Role: FOLLOWER
I20260807 18:32:58.041778 15989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713a109be29644d2b71b7c490979f109" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34125 } }
I20260807 18:32:58.041972 15989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:32:58.042081 15989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:32:58.042209 15989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:32:58.044401 15989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713a109be29644d2b71b7c490979f109" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34125 } }
I20260807 18:32:58.044832 15989 leader_election.cc:302] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 713a109be29644d2b71b7c490979f109; no voters:
I20260807 18:32:58.045374 15989 leader_election.cc:288] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:32:58.045506 15994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:32:58.046806 15994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [term 1 LEADER]: Becoming Leader. State: Replica: 713a109be29644d2b71b7c490979f109, State: Running, Role: LEADER
I20260807 18:32:58.047441 15994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713a109be29644d2b71b7c490979f109" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34125 } }
I20260807 18:32:58.047731 15989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:32:58.050405 15996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 713a109be29644d2b71b7c490979f109. Latest consensus state: current_term: 1 leader_uuid: "713a109be29644d2b71b7c490979f109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713a109be29644d2b71b7c490979f109" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34125 } } }
I20260807 18:32:58.050907 15996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:58.051532 15995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "713a109be29644d2b71b7c490979f109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713a109be29644d2b71b7c490979f109" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34125 } } }
I20260807 18:32:58.051880 15995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109 [sys.catalog]: This master's current role is: LEADER
I20260807 18:32:58.056092 16002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:32:58.060982 16002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:32:58.069110 16002 catalog_manager.cc:1347] Generated new cluster ID: 486ae74cead04e2b8f4beca3413f628c
I20260807 18:32:58.069232 16002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:32:58.079900 16002 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:32:58.081204 16002 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:32:58.086862 16002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109: Generated new TSK 0
I20260807 18:32:58.087690 16002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
Entry for principal kudu/127.27.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab.
I20260807 18:32:58.166384 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34125
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:58.505626 16016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:58.505880 16016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:58.505976 16016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:58.512817 16016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:58.512919 16016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:58.513031 16016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:32:58.522480 16016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.193.keytab
--principal=kudu/127.27.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:58.523705 16016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:58.524951 16016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:58.538005 16024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:58.537976 16022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:58.538113 16021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:58.539072 16016 server_base.cc:1034] running on GCE node
I20260807 18:32:58.539662 16016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:58.540488 16016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:58.541719 16016 hybrid_clock.cc:648] HybridClock initialized: now 1786127578541685 us; error 44 us; skew 500 ppm
I20260807 18:32:58.542018 16016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:58 dist-test-slave-1xrh krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127578, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:58.545189 16016 init.cc:377] Logged in from keytab as kudu/127.27.1.193@KRBTEST.COM (short username kudu)
I20260807 18:32:58.546548 16016 webserver.cc:460] Webserver started at http://127.27.1.193:38035/ using document root <none> and password file <none>
I20260807 18:32:58.547288 16016 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:58.547441 16016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:58.547816 16016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:58.550866 16016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "9068a34ca8314045bf3bc2bb32e7ec64"
format_stamp: "Formatted at 2026-08-07 18:32:58 on dist-test-slave-1xrh"
server_key: "fa509f25adba56f313ce4b22afe988db"
server_key_iv: "53f25ecb2e4d4bbc43f723d765294e78"
server_key_version: "encryptionkey@0"
I20260807 18:32:58.551704 16016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "9068a34ca8314045bf3bc2bb32e7ec64"
format_stamp: "Formatted at 2026-08-07 18:32:58 on dist-test-slave-1xrh"
server_key: "fa509f25adba56f313ce4b22afe988db"
server_key_iv: "53f25ecb2e4d4bbc43f723d765294e78"
server_key_version: "encryptionkey@0"
I20260807 18:32:58.557286 16016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:32:58.560672 16031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:58.562034 16016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 18:32:58.562204 16016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "9068a34ca8314045bf3bc2bb32e7ec64"
format_stamp: "Formatted at 2026-08-07 18:32:58 on dist-test-slave-1xrh"
server_key: "fa509f25adba56f313ce4b22afe988db"
server_key_iv: "53f25ecb2e4d4bbc43f723d765294e78"
server_key_version: "encryptionkey@0"
I20260807 18:32:58.562397 16016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:58.585425 16016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:58.589620 16016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:58.589915 16016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:58.590852 16016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:58.592207 16016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:58.592316 16016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:58.592463 16016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:58.592538 16016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:58.625625 16016 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:35011
I20260807 18:32:58.625684 16151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:35011 every 8 connection(s)
I20260807 18:32:58.626956 16016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:32:58.635352 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 16016
I20260807 18:32:58.635576 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:32:58.636049 27655 external_mini_cluster.cc:1428] Setting key d07ab50f87907cd939e4610885c3a2f1
Aug 07 18:32:58 dist-test-slave-1xrh krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127578, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.193@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:58.648628 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:58.629534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:33759 (local address 127.27.1.254:34125)
0807 18:32:58.630020 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:58.630027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:58.630975 (+ 948us) server_negotiation.cc:407] Connection header received
0807 18:32:58.632855 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:58.632859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:58.632937 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:58.633034 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:58.634424 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:58.635287 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:58.637067 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:58.637504 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:58.637653 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:58.641385 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:58.641394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:58.641402 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:58.641446 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:58.645015 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:58.645744 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:58.645753 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:58.645760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:58.645936 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:58.646401 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:58.646410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:58.646417 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:58.646863 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:58.647043 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:58.647776 (+ 733us) server_negotiation.cc:299] Negotiation successful
0807 18:32:58.648055 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":216,"threads_started":1}
I20260807 18:32:58.649286 16155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:58.630102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34125 (local address 127.27.1.193:33759)
0807 18:32:58.630819 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:58.630857 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:58.632591 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:58.633229 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:58.633236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:58.633738 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:58.634278 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:58.634294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:58.635488 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:58.635492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:58.636893 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:58.636906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:58.637790 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:58.637798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:58.638064 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:58.638760 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:58.638783 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:58.641167 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:58.645156 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:58.645162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:58.645173 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:58.645591 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:58.646068 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:58.646071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:58.646074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:58.646243 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:58.647001 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:58.647006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:58.647377 (+ 371us) client_negotiation.cc:769] Sending connection context
0807 18:32:58.647605 (+ 228us) client_negotiation.cc:240] Negotiation successful
0807 18:32:58.648018 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":283,"threads_started":1}
I20260807 18:32:58.650892 16152 heartbeater.cc:347] Connected to a master server at 127.27.1.254:34125
I20260807 18:32:58.651247 16152 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:58.652135 16152 heartbeater.cc:511] Master 127.27.1.254:34125 requested a full tablet report, sending...
I20260807 18:32:58.654573 15953 ts_manager.cc:194] Registered new tserver with Master: 9068a34ca8314045bf3bc2bb32e7ec64 (127.27.1.193:35011)
I20260807 18:32:58.656796 15953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.193@KRBTEST.COM'} at 127.27.1.193:33759
WARNING: no policy specified for kudu/127.27.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
Entry for principal kudu/127.27.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab.
I20260807 18:32:58.705643 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34125
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:59.049808 16159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:59.050065 16159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:59.050163 16159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:59.057376 16159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:59.057487 16159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:59.057603 16159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:32:59.067386 16159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.194.keytab
--principal=kudu/127.27.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:59.068657 16159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:59.069972 16159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:59.083241 16165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:59.083278 16167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:59.083477 16164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:59.084263 16159 server_base.cc:1034] running on GCE node
I20260807 18:32:59.084842 16159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:59.085722 16159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:59.086956 16159 hybrid_clock.cc:648] HybridClock initialized: now 1786127579086923 us; error 48 us; skew 500 ppm
I20260807 18:32:59.087311 16159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:59 dist-test-slave-1xrh krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127579, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:59.090297 16159 init.cc:377] Logged in from keytab as kudu/127.27.1.194@KRBTEST.COM (short username kudu)
I20260807 18:32:59.091801 16159 webserver.cc:460] Webserver started at http://127.27.1.194:40201/ using document root <none> and password file <none>
I20260807 18:32:59.092408 16159 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:59.092518 16159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:59.092841 16159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:59.095767 16159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "ea0a8be95c45489b8593f5014857288a"
format_stamp: "Formatted at 2026-08-07 18:32:59 on dist-test-slave-1xrh"
server_key: "babf2bf3c9887722f8dfc9e0543c34a8"
server_key_iv: "f22efdc858ce994b6c00c0497b978b77"
server_key_version: "encryptionkey@0"
I20260807 18:32:59.096459 16159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "ea0a8be95c45489b8593f5014857288a"
format_stamp: "Formatted at 2026-08-07 18:32:59 on dist-test-slave-1xrh"
server_key: "babf2bf3c9887722f8dfc9e0543c34a8"
server_key_iv: "f22efdc858ce994b6c00c0497b978b77"
server_key_version: "encryptionkey@0"
I20260807 18:32:59.101468 16159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 18:32:59.104923 16175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:59.106191 16159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 18:32:59.106353 16159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "ea0a8be95c45489b8593f5014857288a"
format_stamp: "Formatted at 2026-08-07 18:32:59 on dist-test-slave-1xrh"
server_key: "babf2bf3c9887722f8dfc9e0543c34a8"
server_key_iv: "f22efdc858ce994b6c00c0497b978b77"
server_key_version: "encryptionkey@0"
I20260807 18:32:59.106570 16159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:59.120419 16159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:59.124365 16159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:59.124652 16159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:59.125594 16159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:59.126853 16159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:59.126926 16159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:59.127070 16159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:59.127189 16159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 18:32:59.161015 16159 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:44915
I20260807 18:32:59.161060 16295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:44915 every 8 connection(s)
I20260807 18:32:59.162335 16159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:32:59.164116 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 16159
I20260807 18:32:59.164363 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:32:59.164990 27655 external_mini_cluster.cc:1428] Setting key 909501d9e3a25d08d2f5e3ca7e161e82
Aug 07 18:32:59 dist-test-slave-1xrh krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127579, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.194@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:59.186362 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.165639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34125 (local address 127.27.1.194:46137)
0807 18:32:59.166406 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:59.166508 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:59.168571 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:59.169249 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:59.169257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:59.169748 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:59.170288 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:59.170306 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.171826 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.171830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:59.173033 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:59.173045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.174135 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.174139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:59.174319 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.175013 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:59.175033 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:59.177434 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:59.181438 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:59.181449 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:59.181469 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:59.182128 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:59.182516 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:59.182523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:59.182529 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:59.182772 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:59.183350 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:59.183361 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:59.183852 (+ 491us) client_negotiation.cc:769] Sending connection context
0807 18:32:59.184160 (+ 308us) client_negotiation.cc:240] Negotiation successful
0807 18:32:59.184752 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":345,"threads_started":1}
I20260807 18:32:59.187721 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.165011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:46137 (local address 127.27.1.254:34125)
0807 18:32:59.165616 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:59.165622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:59.166611 (+ 989us) server_negotiation.cc:407] Connection header received
0807 18:32:59.168834 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:59.168842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:59.168932 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:59.169055 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:59.170465 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.171687 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.173179 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.173925 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.174073 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.177680 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:59.177689 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:59.177696 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:59.177740 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:59.181220 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:59.182258 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:59.182261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:59.182264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:59.182396 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:59.182881 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:59.182884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:59.182886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:59.183219 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:59.183345 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:59.186945 (+ 3600us) server_negotiation.cc:299] Negotiation successful
0807 18:32:59.187247 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":252,"threads_started":1}
I20260807 18:32:59.188901 16296 heartbeater.cc:347] Connected to a master server at 127.27.1.254:34125
I20260807 18:32:59.189262 16296 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:59.190234 16296 heartbeater.cc:511] Master 127.27.1.254:34125 requested a full tablet report, sending...
I20260807 18:32:59.191931 15953 ts_manager.cc:194] Registered new tserver with Master: ea0a8be95c45489b8593f5014857288a (127.27.1.194:44915)
I20260807 18:32:59.192777 15953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.194@KRBTEST.COM'} at 127.27.1.194:46137
WARNING: no policy specified for kudu/127.27.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
Entry for principal kudu/127.27.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab.
I20260807 18:32:59.238838 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34125
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:32:59.593430 16302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:32:59.593698 16302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:32:59.593865 16302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:32:59.601024 16302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:32:59.601128 16302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:32:59.601338 16302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:32:59.610852 16302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.195.keytab
--principal=kudu/127.27.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:34125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:32:59.612102 16302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:32:59.613838 16302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:32:59.626883 16307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:59.626929 16308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:32:59.627846 16310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:32:59.629091 16302 server_base.cc:1034] running on GCE node
I20260807 18:32:59.629662 16302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:32:59.630594 16302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:32:59.631819 16302 hybrid_clock.cc:648] HybridClock initialized: now 1786127579631759 us; error 53 us; skew 500 ppm
I20260807 18:32:59.632145 16302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:32:59 dist-test-slave-1xrh krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127579, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:32:59.635239 16302 init.cc:377] Logged in from keytab as kudu/127.27.1.195@KRBTEST.COM (short username kudu)
I20260807 18:32:59.636778 16302 webserver.cc:460] Webserver started at http://127.27.1.195:41747/ using document root <none> and password file <none>
I20260807 18:32:59.637488 16302 fs_manager.cc:359] Metadata directory not provided
I20260807 18:32:59.637645 16302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:32:59.637982 16302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:32:59.640863 16302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "600440139cb7472da0d1447e58687e63"
format_stamp: "Formatted at 2026-08-07 18:32:59 on dist-test-slave-1xrh"
server_key: "a3695e228f535d88ac1ab0008a7cb661"
server_key_iv: "6a55aa771011759ed55110fc0a2a4b17"
server_key_version: "encryptionkey@0"
I20260807 18:32:59.641644 16302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "600440139cb7472da0d1447e58687e63"
format_stamp: "Formatted at 2026-08-07 18:32:59 on dist-test-slave-1xrh"
server_key: "a3695e228f535d88ac1ab0008a7cb661"
server_key_iv: "6a55aa771011759ed55110fc0a2a4b17"
server_key_version: "encryptionkey@0"
I20260807 18:32:59.646651 16302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 18:32:59.650187 16317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:59.651755 16302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 18:32:59.651955 16302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "600440139cb7472da0d1447e58687e63"
format_stamp: "Formatted at 2026-08-07 18:32:59 on dist-test-slave-1xrh"
server_key: "a3695e228f535d88ac1ab0008a7cb661"
server_key_iv: "6a55aa771011759ed55110fc0a2a4b17"
server_key_version: "encryptionkey@0"
I20260807 18:32:59.652163 16302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:32:59.659643 16152 heartbeater.cc:503] Master 127.27.1.254:34125 was elected leader, sending a full tablet report...
I20260807 18:32:59.670233 16302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:32:59.674099 16302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:32:59.674391 16302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:32:59.675379 16302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:32:59.676699 16302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:32:59.676813 16302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:59.676990 16302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:32:59.677064 16302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:32:59.711907 16302 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:46839
I20260807 18:32:59.711942 16436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:46839 every 8 connection(s)
I20260807 18:32:59.713231 16302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:32:59.719048 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 16302
I20260807 18:32:59.719269 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:32:59.719729 27655 external_mini_cluster.cc:1428] Setting key 89437408a57977a286309a2aa0569c4b
Aug 07 18:32:59 dist-test-slave-1xrh krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127579, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.195@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:32:59.733146 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.715786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:57479 (local address 127.27.1.254:34125)
0807 18:32:59.716278 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:59.716283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:59.717074 (+ 791us) server_negotiation.cc:407] Connection header received
0807 18:32:59.718948 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:59.718953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:59.719046 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:59.719200 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:59.721094 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.722172 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.723893 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.724322 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.724416 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.727746 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:59.727751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:59.727755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:59.727785 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:59.730374 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:59.731033 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:59.731036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:59.731038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:59.731201 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:59.731631 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:59.731633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:59.731635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:59.731884 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:59.731991 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:59.732615 (+ 624us) server_negotiation.cc:299] Negotiation successful
0807 18:32:59.732769 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":246,"threads_started":1}
I20260807 18:32:59.734160 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.716217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:34125 (local address 127.27.1.195:57479)
0807 18:32:59.716909 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:59.716952 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:59.718672 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:59.719492 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:59.719500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:59.720066 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:32:59.720746 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:59.720766 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.722344 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.722348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:59.723739 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:59.723752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.724524 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.724529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:59.724709 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.725328 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:32:59.725354 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:32:59.727562 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:32:59.730502 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:59.730509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:59.730520 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:59.730932 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:59.731312 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:32:59.731315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:32:59.731318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:32:59.731502 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:32:59.731992 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:32:59.731998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:32:59.732294 (+ 296us) client_negotiation.cc:769] Sending connection context
0807 18:32:59.732517 (+ 223us) client_negotiation.cc:240] Negotiation successful
0807 18:32:59.732893 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":279,"threads_started":1}
I20260807 18:32:59.735560 16437 heartbeater.cc:347] Connected to a master server at 127.27.1.254:34125
I20260807 18:32:59.735846 16437 heartbeater.cc:464] Registering TS with master...
I20260807 18:32:59.736644 16437 heartbeater.cc:511] Master 127.27.1.254:34125 requested a full tablet report, sending...
I20260807 18:32:59.738013 15952 ts_manager.cc:194] Registered new tserver with Master: 600440139cb7472da0d1447e58687e63 (127.27.1.195:46839)
I20260807 18:32:59.738871 15952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.1.195@KRBTEST.COM'} at 127.27.1.195:57479
I20260807 18:32:59.747426 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:32:59.765156 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.751876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.27.1.254:34125)
0807 18:32:59.752089 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:59.752098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:59.752429 (+ 331us) server_negotiation.cc:407] Connection header received
0807 18:32:59.752632 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:59.752639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:59.752705 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:59.752805 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:32:59.754197 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.754960 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.756583 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.757191 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.757342 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.759885 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:32:59.759889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:32:59.759892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:32:59.759923 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:32:59.762385 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:59.763056 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:59.763062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:59.763066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:59.763257 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:32:59.763637 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:32:59.763645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:32:59.763648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:32:59.763996 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:32:59.764258 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:59.764604 (+ 346us) server_negotiation.cc:299] Negotiation successful
0807 18:32:59.764849 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 18:32:59.769325 15952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:32:59.776216 15952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:32:59.799909 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.788563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:35011 (local address 127.0.0.1:41846)
0807 18:32:59.789481 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:59.789500 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:59.789625 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:59.792185 (+ 2560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:59.792196 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:59.792252 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:59.793197 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:59.793213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.795191 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.795198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:59.797088 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:59.797103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.798930 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.798936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:59.799151 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.799207 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 18:32:59.799353 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 18:32:59.799547 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":4025,"threads_started":1}
I20260807 18:32:59.799908 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.789192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.27.1.193:35011)
0807 18:32:59.791414 (+ 2222us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:59.791426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:59.791464 (+ 38us) server_negotiation.cc:407] Connection header received
0807 18:32:59.791591 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:59.791603 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:59.791894 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:59.792171 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:59.793462 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.795033 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.797299 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.798751 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.798895 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.799165 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:59.799408 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 18:32:59.799563 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2032,"thread_start_us":1849,"threads_started":1}
I20260807 18:32:59.800976 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.788913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:46839 (local address 127.0.0.1:34330)
0807 18:32:59.793362 (+ 4449us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:59.793379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:59.793498 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:59.794155 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:59.794160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:59.794189 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:59.794581 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:59.794593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.796491 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.796497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:59.798397 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:59.798410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.800393 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.800398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:59.800514 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.800535 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 18:32:59.800605 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 18:32:59.800683 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4263,"thread_start_us":902,"threads_started":1}
I20260807 18:32:59.801222 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.789541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.27.1.195:46839)
0807 18:32:59.790402 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:59.790415 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:59.793642 (+ 3227us) server_negotiation.cc:407] Connection header received
0807 18:32:59.793739 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:59.793746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:59.793960 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:59.794170 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:59.794786 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.796352 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.798559 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.799954 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.800119 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.800326 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:59.800680 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 18:32:59.800854 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":1004,"threads_started":1}
I20260807 18:32:59.802991 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.787878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:44915 (local address 127.0.0.1:60106)
0807 18:32:59.788577 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0807 18:32:59.788627 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 18:32:59.788835 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:32:59.793706 (+ 4871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:32:59.793711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:32:59.793736 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:32:59.794123 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:59.794138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.796283 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.796288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:59.798647 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:32:59.798663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.802173 (+ 3510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.802179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:32:59.802363 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.802396 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 18:32:59.802509 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 18:32:59.802628 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":62,"thread_start_us":287,"threads_started":1}
I20260807 18:32:59.803085 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:32:59.788156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.27.1.194:44915)
0807 18:32:59.793119 (+ 4963us) server_negotiation.cc:206] Beginning negotiation
0807 18:32:59.793127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:32:59.793157 (+ 30us) server_negotiation.cc:407] Connection header received
0807 18:32:59.793253 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:32:59.793260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:32:59.793487 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:32:59.794330 (+ 843us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:32:59.794564 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.796139 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.798814 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:32:59.801921 (+ 3107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:32:59.802126 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:32:59.802254 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 18:32:59.802564 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 18:32:59.802735 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4775,"thread_start_us":4636,"threads_started":1}
I20260807 18:32:59.804198 16085 tablet_service.cc:1467] Processing CreateTablet for tablet 4df2dfb85b47473eb1f1ed011aa81048 (DEFAULT_TABLE table=test-table [id=23dce94de87a4a44818ca8371aa97fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:59.804786 16371 tablet_service.cc:1467] Processing CreateTablet for tablet 4df2dfb85b47473eb1f1ed011aa81048 (DEFAULT_TABLE table=test-table [id=23dce94de87a4a44818ca8371aa97fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:59.805773 16085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df2dfb85b47473eb1f1ed011aa81048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:59.805824 16371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df2dfb85b47473eb1f1ed011aa81048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:59.806376 16227 tablet_service.cc:1467] Processing CreateTablet for tablet 4df2dfb85b47473eb1f1ed011aa81048 (DEFAULT_TABLE table=test-table [id=23dce94de87a4a44818ca8371aa97fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:32:59.807667 16227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df2dfb85b47473eb1f1ed011aa81048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:32:59.817473 16457 tablet_bootstrap.cc:492] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: Bootstrap starting.
I20260807 18:32:59.820732 16458 tablet_bootstrap.cc:492] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: Bootstrap starting.
I20260807 18:32:59.822176 16459 tablet_bootstrap.cc:492] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: Bootstrap starting.
I20260807 18:32:59.822780 16457 tablet_bootstrap.cc:654] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:59.824159 16457 log.cc:824] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:59.824541 16458 tablet_bootstrap.cc:654] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:59.824993 16459 tablet_bootstrap.cc:654] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: Neither blocks nor log segments found. Creating new log.
I20260807 18:32:59.825896 16459 log.cc:824] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:59.826072 16458 log.cc:824] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: Log is configured to *not* fsync() on all Append() calls
I20260807 18:32:59.829239 16459 tablet_bootstrap.cc:492] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: No bootstrap required, opened a new log
I20260807 18:32:59.829299 16458 tablet_bootstrap.cc:492] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: No bootstrap required, opened a new log
I20260807 18:32:59.829299 16457 tablet_bootstrap.cc:492] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: No bootstrap required, opened a new log
I20260807 18:32:59.829537 16459 ts_tablet_manager.cc:1397] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 18:32:59.829560 16457 ts_tablet_manager.cc:1397] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260807 18:32:59.829559 16458 ts_tablet_manager.cc:1397] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 18:32:59.836344 16459 raft_consensus.cc:348] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 } } peers { permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 } } peers { permanent_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35011 } }
I20260807 18:32:59.836725 16459 raft_consensus.cc:374] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:59.836858 16459 raft_consensus.cc:729] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0a8be95c45489b8593f5014857288a, State: Initialized, Role: FOLLOWER
I20260807 18:32:59.837471 16459 consensus_queue.cc:260] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 } } peers { permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 } } peers { permanent_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35011 } }
I20260807 18:32:59.839294 16459 ts_tablet_manager.cc:1428] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260807 18:32:59.839929 16296 heartbeater.cc:503] Master 127.27.1.254:34125 was elected leader, sending a full tablet report...
I20260807 18:32:59.840211 16458 raft_consensus.cc:348] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 } } peers { permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 } } peers { permanent_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35011 } }
I20260807 18:32:59.840638 16458 raft_consensus.cc:374] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:59.840750 16458 raft_consensus.cc:729] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9068a34ca8314045bf3bc2bb32e7ec64, State: Initialized, Role: FOLLOWER
I20260807 18:32:59.840639 16457 raft_consensus.cc:348] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 } } peers { permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 } } peers { permanent_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35011 } }
I20260807 18:32:59.841041 16457 raft_consensus.cc:374] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:32:59.841159 16457 raft_consensus.cc:729] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600440139cb7472da0d1447e58687e63, State: Initialized, Role: FOLLOWER
I20260807 18:32:59.841436 16458 consensus_queue.cc:260] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 } } peers { permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 } } peers { permanent_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35011 } }
I20260807 18:32:59.841720 16457 consensus_queue.cc:260] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 } } peers { permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 } } peers { permanent_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35011 } }
I20260807 18:32:59.843431 16458 ts_tablet_manager.cc:1428] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260807 18:32:59.844882 16457 ts_tablet_manager.cc:1428] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260807 18:32:59.845217 16437 heartbeater.cc:503] Master 127.27.1.254:34125 was elected leader, sending a full tablet report...
W20260807 18:32:59.879460 16153 tablet.cc:2369] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:32:59.914291 16297 tablet.cc:2369] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:32:59.964913 16438 tablet.cc:2369] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:33:00.023320 16464 raft_consensus.cc:482] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:33:00.023605 16464 raft_consensus.cc:504] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 } } peers { permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 } } peers { permanent_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35011 } }
I20260807 18:33:00.025564 16464 leader_election.cc:288] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea0a8be95c45489b8593f5014857288a (127.27.1.194:44915), 600440139cb7472da0d1447e58687e63 (127.27.1.195:46839)
I20260807 18:33:00.033941 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:00.025944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:52927 (local address 127.27.1.194:44915)
0807 18:33:00.026142 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:00.026147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:00.026843 (+ 696us) server_negotiation.cc:407] Connection header received
0807 18:33:00.027079 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:00.027084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:00.027199 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:00.027312 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:33:00.028012 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.029037 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.031646 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.032919 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.033114 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:00.033217 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:00.033522 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 18:33:00.033674 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:33:00.033964 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:00.027186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:53705 (local address 127.27.1.195:46839)
0807 18:33:00.027418 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:00.027424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:00.027661 (+ 237us) server_negotiation.cc:407] Connection header received
0807 18:33:00.027831 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:00.027836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:00.027897 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:00.028006 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:33:00.028488 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.029433 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.031857 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.033095 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.033241 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:00.033303 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:00.033621 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 18:33:00.033740 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 18:33:00.034097 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:00.025943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:44915 (local address 127.27.1.193:52927)
0807 18:33:00.026715 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:00.026737 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:00.026878 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:00.027463 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:00.027467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:00.027495 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:33:00.027874 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:00.027884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.029168 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.029171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:00.031481 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:00.031497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.033131 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.033135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:00.033353 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:00.033390 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 18:33:00.033488 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 18:33:00.033634 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":96,"thread_start_us":425,"threads_started":1}
I20260807 18:33:00.034307 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:00.026833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:46839 (local address 127.27.1.193:53705)
0807 18:33:00.027556 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:00.027579 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:00.027711 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:00.028047 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:00.028050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:00.028071 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:33:00.028346 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:00.028353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.029571 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.029577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:00.031711 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:00.031720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.033294 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.033300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:00.033449 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:00.033500 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 18:33:00.033575 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 18:33:00.033675 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":21,"thread_start_us":392,"threads_started":1}
I20260807 18:33:00.035296 16248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df2dfb85b47473eb1f1ed011aa81048" candidate_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0a8be95c45489b8593f5014857288a" is_pre_election: true
I20260807 18:33:00.035339 16391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df2dfb85b47473eb1f1ed011aa81048" candidate_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600440139cb7472da0d1447e58687e63" is_pre_election: true
I20260807 18:33:00.035821 16391 raft_consensus.cc:2454] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9068a34ca8314045bf3bc2bb32e7ec64 in term 0.
I20260807 18:33:00.035816 16248 raft_consensus.cc:2454] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9068a34ca8314045bf3bc2bb32e7ec64 in term 0.
I20260807 18:33:00.036484 16032 leader_election.cc:302] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 600440139cb7472da0d1447e58687e63, 9068a34ca8314045bf3bc2bb32e7ec64; no voters:
I20260807 18:33:00.036875 16464 raft_consensus.cc:2781] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:33:00.037019 16464 raft_consensus.cc:482] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:33:00.037143 16464 raft_consensus.cc:3037] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:00.039496 16464 raft_consensus.cc:504] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 } } peers { permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 } } peers { permanent_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35011 } }
I20260807 18:33:00.040062 16464 leader_election.cc:288] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [CANDIDATE]: Term 1 election: Requested vote from peers ea0a8be95c45489b8593f5014857288a (127.27.1.194:44915), 600440139cb7472da0d1447e58687e63 (127.27.1.195:46839)
I20260807 18:33:00.040607 16248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df2dfb85b47473eb1f1ed011aa81048" candidate_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0a8be95c45489b8593f5014857288a"
I20260807 18:33:00.040719 16391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df2dfb85b47473eb1f1ed011aa81048" candidate_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600440139cb7472da0d1447e58687e63"
I20260807 18:33:00.040961 16248 raft_consensus.cc:3037] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:00.040957 16391 raft_consensus.cc:3037] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:00.044593 16248 raft_consensus.cc:2454] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9068a34ca8314045bf3bc2bb32e7ec64 in term 1.
I20260807 18:33:00.044593 16391 raft_consensus.cc:2454] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9068a34ca8314045bf3bc2bb32e7ec64 in term 1.
I20260807 18:33:00.045075 16032 leader_election.cc:302] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9068a34ca8314045bf3bc2bb32e7ec64, ea0a8be95c45489b8593f5014857288a; no voters:
I20260807 18:33:00.045362 16464 raft_consensus.cc:2781] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:00.045816 16464 raft_consensus.cc:686] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 1 LEADER]: Becoming Leader. State: Replica: 9068a34ca8314045bf3bc2bb32e7ec64, State: Running, Role: LEADER
I20260807 18:33:00.046360 16464 consensus_queue.cc:237] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 } } peers { permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 } } peers { permanent_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35011 } }
I20260807 18:33:00.051535 15952 catalog_manager.cc:5495] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9068a34ca8314045bf3bc2bb32e7ec64 (127.27.1.193). New cstate: current_term: 1 leader_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9068a34ca8314045bf3bc2bb32e7ec64" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 35011 } health_report { overall_health: HEALTHY } } }
I20260807 18:33:00.096715 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:00.089925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.27.1.193:35011)
0807 18:33:00.090154 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:00.090159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:00.090253 (+ 94us) server_negotiation.cc:407] Connection header received
0807 18:33:00.090486 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:00.090489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:00.090572 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:00.090700 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:00.091235 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.091936 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.094365 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.094768 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.094866 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:00.095594 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:00.096137 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:00.096255 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:00.096397 (+ 142us) server_negotiation.cc:299] Negotiation successful
0807 18:33:00.096468 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 18:33:00.105265 16248 raft_consensus.cc:1261] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a [term 1 FOLLOWER]: Refusing update from remote peer 9068a34ca8314045bf3bc2bb32e7ec64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:33:00.105482 16391 raft_consensus.cc:1261] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63 [term 1 FOLLOWER]: Refusing update from remote peer 9068a34ca8314045bf3bc2bb32e7ec64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:33:00.106382 16469 consensus_queue.cc:1035] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0a8be95c45489b8593f5014857288a" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:33:00.106801 16464 consensus_queue.cc:1035] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "600440139cb7472da0d1447e58687e63" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:33:00.125310 16472 mvcc.cc:205] Tried to move back new op lower bound from 7315978568100720640 to 7315978567874117632. Current Snapshot: MvccSnapshot[applied={T|T < 7315978568100720640}]
I20260807 18:33:00.127172 16473 mvcc.cc:205] Tried to move back new op lower bound from 7315978568100720640 to 7315978567874117632. Current Snapshot: MvccSnapshot[applied={T|T < 7315978568100720640}]
I20260807 18:33:00.147040 15952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:33:00.147425 15952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:33:00.152436 15952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 713a109be29644d2b71b7c490979f109: Sending DeleteTablet for 3 replicas of tablet 4df2dfb85b47473eb1f1ed011aa81048
I20260807 18:33:00.153784 16085 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df2dfb85b47473eb1f1ed011aa81048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:33:00 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:41846
I20260807 18:33:00.154068 16371 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df2dfb85b47473eb1f1ed011aa81048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:33:00 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:34330
I20260807 18:33:00.154560 16227 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df2dfb85b47473eb1f1ed011aa81048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:33:00 UTC) from {username='kudu', principal='kudu/127.27.1.254@KRBTEST.COM'} at 127.0.0.1:60106
I20260807 18:33:00.169696 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 16016
I20260807 18:33:00.175632 16484 tablet_replica.cc:330] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: stopping tablet replica
I20260807 18:33:00.175918 16482 tablet_replica.cc:330] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: stopping tablet replica
I20260807 18:33:00.176102 16484 raft_consensus.cc:2229] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:33:00.176321 16482 raft_consensus.cc:2229] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:33:00.176635 16484 raft_consensus.cc:2258] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:33:00.177292 16482 raft_consensus.cc:2258] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:33:00.179230 16484 ts_tablet_manager.cc:1905] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:33:00.179538 16482 ts_tablet_manager.cc:1905] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:33:00.179564 16483 tablet_replica.cc:330] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: stopping tablet replica
I20260807 18:33:00.179944 16483 raft_consensus.cc:2229] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:33:00.180444 16483 raft_consensus.cc:2258] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:33:00.182375 16483 ts_tablet_manager.cc:1905] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:33:00.187054 16482 ts_tablet_manager.cc:1918] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:33:00.187263 16482 log.cc:1196] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-0/wal/wals/4df2dfb85b47473eb1f1ed011aa81048
I20260807 18:33:00.187891 16482 ts_tablet_manager.cc:1939] T 4df2dfb85b47473eb1f1ed011aa81048 P 9068a34ca8314045bf3bc2bb32e7ec64: Deleting consensus metadata
I20260807 18:33:00.190011 15939 catalog_manager.cc:4841] TS 9068a34ca8314045bf3bc2bb32e7ec64 (127.27.1.193:35011): tablet 4df2dfb85b47473eb1f1ed011aa81048 (table test-table [id=23dce94de87a4a44818ca8371aa97fc2]) successfully deleted
I20260807 18:33:00.193025 16484 ts_tablet_manager.cc:1918] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:33:00.193200 16484 log.cc:1196] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/4df2dfb85b47473eb1f1ed011aa81048
I20260807 18:33:00.193769 16484 ts_tablet_manager.cc:1939] T 4df2dfb85b47473eb1f1ed011aa81048 P ea0a8be95c45489b8593f5014857288a: Deleting consensus metadata
I20260807 18:33:00.195624 15937 catalog_manager.cc:4841] TS ea0a8be95c45489b8593f5014857288a (127.27.1.194:44915): tablet 4df2dfb85b47473eb1f1ed011aa81048 (table test-table [id=23dce94de87a4a44818ca8371aa97fc2]) successfully deleted
Aug 07 18:33:00 dist-test-slave-1xrh krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.193@KRBTEST.COM
I20260807 18:33:00.212114 16483 ts_tablet_manager.cc:1918] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:33:00.212401 16483 log.cc:1196] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/4df2dfb85b47473eb1f1ed011aa81048
I20260807 18:33:00.212870 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:00.171602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.27.1.193:35011)
0807 18:33:00.171814 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:00.171821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:00.190822 (+ 19001us) server_negotiation.cc:407] Connection header received
0807 18:33:00.191033 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:00.191067 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:00.191218 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:00.191361 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:00.193585 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.195016 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.197050 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.197780 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.197911 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:00.203596 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:00.203614 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:00.203635 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:00.203704 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:00.208643 (+ 4939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:00.209819 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:00.209834 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:00.209840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:00.210028 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:00.210527 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:00.210533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:00.210536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:00.211106 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:00.211414 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:00.212119 (+ 705us) server_negotiation.cc:299] Negotiation successful
0807 18:33:00.212375 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 18:33:00.213248 16483 ts_tablet_manager.cc:1939] T 4df2dfb85b47473eb1f1ed011aa81048 P 600440139cb7472da0d1447e58687e63: Deleting consensus metadata
I20260807 18:33:00.214375 16147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:00.215436 15937 catalog_manager.cc:4841] TS 600440139cb7472da0d1447e58687e63 (127.27.1.195:46839): tablet 4df2dfb85b47473eb1f1ed011aa81048 (table test-table [id=23dce94de87a4a44818ca8371aa97fc2]) successfully deleted
I20260807 18:33:00.362821 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 16016
I20260807 18:33:00.395637 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 16159
Aug 07 18:33:00 dist-test-slave-1xrh krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.194@KRBTEST.COM
I20260807 18:33:00.411957 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:00.396638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.27.1.194:44915)
0807 18:33:00.396847 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:00.396856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:00.396878 (+ 22us) server_negotiation.cc:407] Connection header received
0807 18:33:00.396954 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:00.396958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:00.397030 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:00.397137 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:00.398475 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.399264 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.400512 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.400908 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.401042 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:00.405652 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:00.405667 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:00.405683 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:00.405721 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:00.408680 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:00.409352 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:00.409359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:00.409362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:00.409501 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:00.409932 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:00.409938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:00.409941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:00.410379 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:00.410737 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:00.411113 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 18:33:00.411452 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 18:33:00.412734 16291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:00.503968 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 16159
I20260807 18:33:00.525996 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 16302
Aug 07 18:33:00 dist-test-slave-1xrh krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.195@KRBTEST.COM
I20260807 18:33:00.541188 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:00.526858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.27.1.195:46839)
0807 18:33:00.527045 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:00.527050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:00.527070 (+ 20us) server_negotiation.cc:407] Connection header received
0807 18:33:00.527279 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:00.527283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:00.527353 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:00.527451 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:00.528732 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.529422 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.530680 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:00.531058 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:00.531238 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:00.535584 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:00.535600 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:00.535618 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:00.535658 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:00.538704 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:00.539443 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:00.539450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:00.539453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:00.539614 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:00.539946 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:00.539949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:00.539951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:00.540286 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:00.540497 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:00.540766 (+ 269us) server_negotiation.cc:299] Negotiation successful
0807 18:33:00.540921 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 18:33:00.541836 16432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:00.633167 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 16302
I20260807 18:33:00.653671 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 15920
I20260807 18:33:00.654875 15984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:00.766821 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 15920
2026-08-07T18:33:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4691 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T18:33:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:00Z Disabled control of system clock
I20260807 18:33:00.860239 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44647
--webserver_interface=127.27.1.254
--webserver_port=0
--builtin_ntp_servers=127.27.1.212:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:44647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 18:33:01.209807 16494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:01.210099 16494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:01.210278 16494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:01.217386 16494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:01.217527 16494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:01.217602 16494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:01.217711 16494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:01.217799 16494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:01.226692 16494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:44647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:44647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:01.227953 16494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:01.229725 16494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:01.241386 16500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:01.241423 16501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:01.241444 16503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:01.242640 16494 server_base.cc:1034] running on GCE node
I20260807 18:33:01.243443 16494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:01.244760 16494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:01.245982 16494 hybrid_clock.cc:648] HybridClock initialized: now 1786127581245941 us; error 55 us; skew 500 ppm
I20260807 18:33:01.246341 16494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:33:01.259950 16494 webserver.cc:460] Webserver started at http://127.27.1.254:36841/ using document root <none> and password file <none>
I20260807 18:33:01.260746 16494 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:01.260897 16494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:01.261245 16494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:01.264277 16494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "91740e1bde64463484524c7139d22a93"
format_stamp: "Formatted at 2026-08-07 18:33:01 on dist-test-slave-1xrh"
server_key: "634062f9bd75582d8b467744d32ffa51"
server_key_iv: "74fdf9eb9a3a21b75b71a8251f8cd978"
server_key_version: "encryptionkey@0"
I20260807 18:33:01.265125 16494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "91740e1bde64463484524c7139d22a93"
format_stamp: "Formatted at 2026-08-07 18:33:01 on dist-test-slave-1xrh"
server_key: "634062f9bd75582d8b467744d32ffa51"
server_key_iv: "74fdf9eb9a3a21b75b71a8251f8cd978"
server_key_version: "encryptionkey@0"
I20260807 18:33:01.270248 16494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 18:33:01.273588 16509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:01.274796 16494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 18:33:01.274982 16494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "91740e1bde64463484524c7139d22a93"
format_stamp: "Formatted at 2026-08-07 18:33:01 on dist-test-slave-1xrh"
server_key: "634062f9bd75582d8b467744d32ffa51"
server_key_iv: "74fdf9eb9a3a21b75b71a8251f8cd978"
server_key_version: "encryptionkey@0"
I20260807 18:33:01.275250 16494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:01.322402 16494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:01.323395 16494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:01.323704 16494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:01.342576 16494 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:44647
I20260807 18:33:01.342613 16562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:44647 every 8 connection(s)
I20260807 18:33:01.343972 16494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:01.347646 16563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:01.351356 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 16494
I20260807 18:33:01.351547 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:01.352118 27655 external_mini_cluster.cc:1428] Setting key 496a48d3975f7207a16c5d6ef905d07b
I20260807 18:33:01.360363 16563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93: Bootstrap starting.
I20260807 18:33:01.363539 16563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:01.364712 16563 log.cc:824] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:01.365808 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:01.354207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.27.1.254:44647)
0807 18:33:01.355219 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:01.355255 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:01.355299 (+ 44us) server_negotiation.cc:407] Connection header received
0807 18:33:01.356936 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:01.356958 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:01.357411 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:01.357815 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:01.358400 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:01.359720 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:01.361147 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:01.361874 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:01.362081 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:01.363245 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:01.363252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:01.363266 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:01.363870 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:01.363955 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:01.363978 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:01.364103 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:01.364302 (+ 199us) server_negotiation.cc:299] Negotiation successful
0807 18:33:01.364600 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":354,"threads_started":1}
I20260807 18:33:01.367682 16563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93: No bootstrap required, opened a new log
I20260807 18:33:01.374701 16563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91740e1bde64463484524c7139d22a93" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44647 } }
I20260807 18:33:01.375059 16563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:01.375240 16563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91740e1bde64463484524c7139d22a93, State: Initialized, Role: FOLLOWER
I20260807 18:33:01.375778 16563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91740e1bde64463484524c7139d22a93" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44647 } }
I20260807 18:33:01.376021 16563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:01.376179 16563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:01.376355 16563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:01.378480 16563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91740e1bde64463484524c7139d22a93" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44647 } }
I20260807 18:33:01.378981 16563 leader_election.cc:302] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91740e1bde64463484524c7139d22a93; no voters:
I20260807 18:33:01.379648 16563 leader_election.cc:288] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:01.379876 16568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:01.381255 16568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [term 1 LEADER]: Becoming Leader. State: Replica: 91740e1bde64463484524c7139d22a93, State: Running, Role: LEADER
I20260807 18:33:01.381587 16563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:01.381902 16568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91740e1bde64463484524c7139d22a93" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44647 } }
I20260807 18:33:01.384974 16569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91740e1bde64463484524c7139d22a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91740e1bde64463484524c7139d22a93" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44647 } } }
I20260807 18:33:01.385442 16569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:01.385324 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91740e1bde64463484524c7139d22a93. Latest consensus state: current_term: 1 leader_uuid: "91740e1bde64463484524c7139d22a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91740e1bde64463484524c7139d22a93" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 44647 } } }
I20260807 18:33:01.385710 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:01.388104 16578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:01.392627 16578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:01.400689 16578 catalog_manager.cc:1347] Generated new cluster ID: d0c9ad57232745c7a6cdfe77e4fca3e9
I20260807 18:33:01.400794 16578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:01.438901 16578 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:01.440209 16578 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:01.447857 16578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93: Generated new TSK 0
I20260807 18:33:01.448855 16578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:01.464176 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.193:0
--local_ip_for_outbound_sockets=127.27.1.193
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44647
--builtin_ntp_servers=127.27.1.212:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 18:33:01.836462 16587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:01.836795 16587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:01.836961 16587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:01.844219 16587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:01.844316 16587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:01.844420 16587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.193
I20260807 18:33:01.853607 16587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.1.193
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.193
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:01.854948 16587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:01.856803 16587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:01.870774 16592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:01.870963 16595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:01.871069 16593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:01.873133 16587 server_base.cc:1034] running on GCE node
I20260807 18:33:01.873850 16587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:01.875051 16587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:01.876368 16587 hybrid_clock.cc:648] HybridClock initialized: now 1786127581876311 us; error 48 us; skew 500 ppm
I20260807 18:33:01.876648 16587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:33:01.878824 16587 webserver.cc:460] Webserver started at http://127.27.1.193:41095/ using document root <none> and password file <none>
I20260807 18:33:01.879616 16587 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:01.879710 16587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:01.880033 16587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:01.882910 16587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/data/instance:
uuid: "252b185b1a6f4d2798b3c1479168078a"
format_stamp: "Formatted at 2026-08-07 18:33:01 on dist-test-slave-1xrh"
server_key: "ba1f5ac5fc020b7ed90c477f9f50c214"
server_key_iv: "a5086b6badd4bfd1368b89f6bc07e521"
server_key_version: "encryptionkey@0"
I20260807 18:33:01.883841 16587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance:
uuid: "252b185b1a6f4d2798b3c1479168078a"
format_stamp: "Formatted at 2026-08-07 18:33:01 on dist-test-slave-1xrh"
server_key: "ba1f5ac5fc020b7ed90c477f9f50c214"
server_key_iv: "a5086b6badd4bfd1368b89f6bc07e521"
server_key_version: "encryptionkey@0"
I20260807 18:33:01.889178 16587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:33:01.892544 16601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:01.893801 16587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:33:01.893983 16587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/wal
uuid: "252b185b1a6f4d2798b3c1479168078a"
format_stamp: "Formatted at 2026-08-07 18:33:01 on dist-test-slave-1xrh"
server_key: "ba1f5ac5fc020b7ed90c477f9f50c214"
server_key_iv: "a5086b6badd4bfd1368b89f6bc07e521"
server_key_version: "encryptionkey@0"
I20260807 18:33:01.894220 16587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:01.913040 16587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:01.913956 16587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:01.914284 16587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:01.915400 16587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:33:01.916925 16587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:33:01.917065 16587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:01.917212 16587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:33:01.917304 16587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:01.950935 16587 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.193:36369
I20260807 18:33:01.951025 16718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.193:36369 every 8 connection(s)
I20260807 18:33:01.952363 16587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/data/info.pb
I20260807 18:33:01.956786 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 16587
I20260807 18:33:01.956970 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-0/wal/instance
I20260807 18:33:01.957557 27655 external_mini_cluster.cc:1428] Setting key 903570efd6282154f3266d55b57ae83e
I20260807 18:33:01.960743 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.194:0
--local_ip_for_outbound_sockets=127.27.1.194
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44647
--builtin_ntp_servers=127.27.1.212:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 18:33:01.966835 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:01.954935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.193:34689 (local address 127.27.1.254:44647)
0807 18:33:01.955470 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:01.955474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:01.956691 (+ 1217us) server_negotiation.cc:407] Connection header received
0807 18:33:01.959102 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:01.959107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:01.959206 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:01.959326 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:01.960711 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:01.961507 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:01.963995 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:01.964396 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:01.964496 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:01.966103 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:01.966108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:01.966111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:01.966129 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:01.966169 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:01.966181 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:01.966247 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:01.966494 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 18:33:01.966579 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":299,"threads_started":1}
I20260807 18:33:01.968300 16722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:01.955508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44647 (local address 127.27.1.193:34689)
0807 18:33:01.956450 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:01.956520 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:01.958834 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:01.959579 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:01.959609 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:01.959716 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:01.960522 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:01.960550 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:01.961679 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:01.961688 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:01.963853 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:01.963872 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:01.964628 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:01.964637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:01.964970 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:01.965738 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:33:01.965743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:01.965794 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:33:01.965898 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:01.966274 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:01.966288 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:01.966298 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 18:33:01.966478 (+ 180us) client_negotiation.cc:240] Negotiation successful
0807 18:33:01.966891 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":279,"threads_started":1}
I20260807 18:33:01.969888 16719 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44647
I20260807 18:33:01.970194 16719 heartbeater.cc:464] Registering TS with master...
I20260807 18:33:01.971148 16719 heartbeater.cc:511] Master 127.27.1.254:44647 requested a full tablet report, sending...
I20260807 18:33:01.973234 16526 ts_manager.cc:194] Registered new tserver with Master: 252b185b1a6f4d2798b3c1479168078a (127.27.1.193:36369)
I20260807 18:33:01.975457 16526 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.193:34689
W20260807 18:33:02.300901 16723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:02.301180 16723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:02.301266 16723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:02.308385 16723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:02.308490 16723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:02.308600 16723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.194
I20260807 18:33:02.317812 16723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.1.194
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.194
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:02.319212 16723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:02.320995 16723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:02.334530 16728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:02.334558 16731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:02.334640 16729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:02.336112 16723 server_base.cc:1034] running on GCE node
I20260807 18:33:02.336787 16723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:02.337760 16723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:02.338999 16723 hybrid_clock.cc:648] HybridClock initialized: now 1786127582338935 us; error 53 us; skew 500 ppm
I20260807 18:33:02.339357 16723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:33:02.341583 16723 webserver.cc:460] Webserver started at http://127.27.1.194:36491/ using document root <none> and password file <none>
I20260807 18:33:02.342314 16723 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:02.342449 16723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:02.342847 16723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:02.345839 16723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/data/instance:
uuid: "b6e331f2c2c54792b70f5f1be2d81d44"
format_stamp: "Formatted at 2026-08-07 18:33:02 on dist-test-slave-1xrh"
server_key: "99bd791c98a837ca19761fbc8d5dac17"
server_key_iv: "d47cfae6da7f281b58a59b32797d7f96"
server_key_version: "encryptionkey@0"
I20260807 18:33:02.346659 16723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance:
uuid: "b6e331f2c2c54792b70f5f1be2d81d44"
format_stamp: "Formatted at 2026-08-07 18:33:02 on dist-test-slave-1xrh"
server_key: "99bd791c98a837ca19761fbc8d5dac17"
server_key_iv: "d47cfae6da7f281b58a59b32797d7f96"
server_key_version: "encryptionkey@0"
I20260807 18:33:02.351850 16723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 18:33:02.355408 16738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:02.356813 16723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:33:02.357000 16723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/wal
uuid: "b6e331f2c2c54792b70f5f1be2d81d44"
format_stamp: "Formatted at 2026-08-07 18:33:02 on dist-test-slave-1xrh"
server_key: "99bd791c98a837ca19761fbc8d5dac17"
server_key_iv: "d47cfae6da7f281b58a59b32797d7f96"
server_key_version: "encryptionkey@0"
I20260807 18:33:02.357219 16723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:02.402602 16723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:02.403644 16723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:02.403993 16723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:02.405037 16723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:33:02.406579 16723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:33:02.406708 16723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:02.406853 16723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:33:02.406945 16723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:02.439822 16723 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.194:40145
I20260807 18:33:02.439853 16858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.194:40145 every 8 connection(s)
I20260807 18:33:02.441174 16723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/data/info.pb
I20260807 18:33:02.444780 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 16723
I20260807 18:33:02.444976 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/wal/instance
I20260807 18:33:02.445559 27655 external_mini_cluster.cc:1428] Setting key b3975336b2821de0335c3596a777863d
I20260807 18:33:02.448625 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.1.195:0
--local_ip_for_outbound_sockets=127.27.1.195
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44647
--builtin_ntp_servers=127.27.1.212:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 18:33:02.454823 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.443857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.194:52897 (local address 127.27.1.254:44647)
0807 18:33:02.444051 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:02.444056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:02.445501 (+ 1445us) server_negotiation.cc:407] Connection header received
0807 18:33:02.447577 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:02.447581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:02.447671 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:02.447771 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:02.449042 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.450217 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.451910 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.452318 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.452461 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.453864 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:02.453869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:02.453872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:02.453890 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:02.453924 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:02.453935 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:02.454038 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:02.454366 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 18:33:02.454472 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:33:02.455997 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.444248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44647 (local address 127.27.1.194:52897)
0807 18:33:02.445335 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:02.445372 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:02.447275 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:02.448088 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:02.448102 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:02.448227 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:02.448865 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:02.448884 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.450386 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.450390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:02.451753 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:02.451765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.452550 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.452558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:02.452796 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.453517 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:33:02.453520 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:02.453564 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:33:02.453657 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:02.454153 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:02.454162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:02.454168 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 18:33:02.454350 (+ 182us) client_negotiation.cc:240] Negotiation successful
0807 18:33:02.454732 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":366,"threads_started":1}
I20260807 18:33:02.457566 16859 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44647
I20260807 18:33:02.457870 16859 heartbeater.cc:464] Registering TS with master...
I20260807 18:33:02.458622 16859 heartbeater.cc:511] Master 127.27.1.254:44647 requested a full tablet report, sending...
I20260807 18:33:02.460201 16526 ts_manager.cc:194] Registered new tserver with Master: b6e331f2c2c54792b70f5f1be2d81d44 (127.27.1.194:40145)
I20260807 18:33:02.461453 16526 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.194:52897
W20260807 18:33:02.806571 16862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:02.806847 16862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:02.806944 16862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:02.814164 16862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:02.814252 16862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:02.814356 16862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.1.195
I20260807 18:33:02.823247 16862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.1.195
--webserver_port=0
--tserver_master_addrs=127.27.1.254:44647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.1.195
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:02.824441 16862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:02.826175 16862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:02.839103 16867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:02.839099 16868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:02.839440 16870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:02.840173 16862 server_base.cc:1034] running on GCE node
I20260807 18:33:02.840764 16862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:02.841717 16862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:02.842972 16862 hybrid_clock.cc:648] HybridClock initialized: now 1786127582842933 us; error 51 us; skew 500 ppm
I20260807 18:33:02.843348 16862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:33:02.845628 16862 webserver.cc:460] Webserver started at http://127.27.1.195:37151/ using document root <none> and password file <none>
I20260807 18:33:02.846328 16862 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:02.846465 16862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:02.846788 16862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:02.849632 16862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/data/instance:
uuid: "e38d0c2b35c14639a063457d0c46ae51"
format_stamp: "Formatted at 2026-08-07 18:33:02 on dist-test-slave-1xrh"
server_key: "c439d1eb972f73a8eb381917a6087aaa"
server_key_iv: "c8e94d934ac8bd73b2ac4d442c106f30"
server_key_version: "encryptionkey@0"
I20260807 18:33:02.850437 16862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance:
uuid: "e38d0c2b35c14639a063457d0c46ae51"
format_stamp: "Formatted at 2026-08-07 18:33:02 on dist-test-slave-1xrh"
server_key: "c439d1eb972f73a8eb381917a6087aaa"
server_key_iv: "c8e94d934ac8bd73b2ac4d442c106f30"
server_key_version: "encryptionkey@0"
I20260807 18:33:02.855719 16862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 18:33:02.859107 16876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:02.860445 16862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:33:02.860620 16862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/wal
uuid: "e38d0c2b35c14639a063457d0c46ae51"
format_stamp: "Formatted at 2026-08-07 18:33:02 on dist-test-slave-1xrh"
server_key: "c439d1eb972f73a8eb381917a6087aaa"
server_key_iv: "c8e94d934ac8bd73b2ac4d442c106f30"
server_key_version: "encryptionkey@0"
I20260807 18:33:02.860831 16862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:02.875211 16862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:02.876297 16862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:02.876645 16862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:02.877872 16862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:33:02.879598 16862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:33:02.879724 16862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:02.879837 16862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:33:02.879951 16862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:02.913575 16862 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.195:43731
I20260807 18:33:02.913661 16994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.195:43731 every 8 connection(s)
I20260807 18:33:02.914942 16862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/data/info.pb
I20260807 18:33:02.922456 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 16862
I20260807 18:33:02.922633 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/wal/instance
I20260807 18:33:02.923225 27655 external_mini_cluster.cc:1428] Setting key ee13fbc1bd055982c112333d8c225080
I20260807 18:33:02.927891 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.917633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:52119 (local address 127.27.1.254:44647)
0807 18:33:02.917829 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:02.917834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:02.919174 (+ 1340us) server_negotiation.cc:407] Connection header received
0807 18:33:02.920877 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:02.920881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:02.920950 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:02.921040 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:02.922006 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.922861 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.925014 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.925686 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.925810 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.927223 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:02.927228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:02.927231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:02.927249 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:02.927278 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:02.927290 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:02.927362 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:02.927573 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 18:33:02.927647 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 18:33:02.928795 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.918115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:44647 (local address 127.27.1.195:52119)
0807 18:33:02.918972 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:02.919014 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:02.920634 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:02.921247 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:02.921256 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:02.921335 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:02.921860 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:02.921875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.923019 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.923034 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:02.924855 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:02.924877 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.925938 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.925948 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:02.926229 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.926884 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 18:33:02.926887 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:02.926922 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 18:33:02.926991 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:02.927380 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:02.927389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:02.927394 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 18:33:02.927547 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 18:33:02.927813 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":283,"threads_started":1}
I20260807 18:33:02.930095 16995 heartbeater.cc:347] Connected to a master server at 127.27.1.254:44647
I20260807 18:33:02.930384 16995 heartbeater.cc:464] Registering TS with master...
I20260807 18:33:02.931114 16995 heartbeater.cc:511] Master 127.27.1.254:44647 requested a full tablet report, sending...
I20260807 18:33:02.932497 16526 ts_manager.cc:194] Registered new tserver with Master: e38d0c2b35c14639a063457d0c46ae51 (127.27.1.195:43731)
I20260807 18:33:02.933256 16526 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.1.195:52119
I20260807 18:33:02.938146 27655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:33:02.947706 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.942123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.27.1.254:44647)
0807 18:33:02.942362 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:02.942367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:02.942800 (+ 433us) server_negotiation.cc:407] Connection header received
0807 18:33:02.942990 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:02.942993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:02.943073 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:02.943218 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:02.943669 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.944344 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.945607 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.945987 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.946083 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.947099 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:02.947103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:02.947106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:02.947123 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:02.947205 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:02.947217 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:02.947286 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:02.947409 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 18:33:02.947477 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:33:02.951383 16526 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:33:02.958359 16526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:33:02.978415 16719 heartbeater.cc:503] Master 127.27.1.254:44647 was elected leader, sending a full tablet report...
I20260807 18:33:02.991289 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.970512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.27.1.195:43731)
0807 18:33:02.981297 (+ 10785us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:02.981311 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:02.982730 (+ 1419us) server_negotiation.cc:407] Connection header received
0807 18:33:02.983193 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:02.983202 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:02.983449 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:02.983634 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:33:02.985081 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.986671 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.989016 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.990066 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.990226 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.990537 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:02.990741 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 18:33:02.990901 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10519,"thread_start_us":10798,"threads_started":1}
I20260807 18:33:02.991377 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.970187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.195:43731 (local address 127.0.0.1:37120)
0807 18:33:02.982560 (+ 12373us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:02.982592 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:02.982904 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:02.983918 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:02.983925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:02.983966 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:33:02.984564 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:02.984578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.986793 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.986800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:02.988778 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:02.988791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.990271 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.990278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:02.990476 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.990532 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 18:33:02.990690 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 18:33:02.990875 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12039,"mutex_wait_us":6,"spinlock_wait_cycles":15616,"thread_start_us":11866,"threads_started":1}
I20260807 18:33:02.992722 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.970399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:40145 (local address 127.0.0.1:39650)
0807 18:33:02.982536 (+ 12137us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:02.982593 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:02.982904 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:02.983917 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:02.983924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:02.983964 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:33:02.984564 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:02.984578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.986508 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.986515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:02.989187 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:02.989203 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.991911 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.991920 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:02.992117 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.992141 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 18:33:02.992250 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 18:33:02.992369 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11889,"mutex_wait_us":288,"spinlock_wait_cycles":8064,"thread_start_us":11749,"threads_started":1}
I20260807 18:33:02.992782 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.970647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.27.1.194:40145)
0807 18:33:02.982000 (+ 11353us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:02.982011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:02.982731 (+ 720us) server_negotiation.cc:407] Connection header received
0807 18:33:02.983196 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:02.983203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:02.983444 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:02.983634 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:33:02.984818 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.986327 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.989399 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.990834 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.990964 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.991199 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:02.992305 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0807 18:33:02.992462 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11191,"thread_start_us":11086,"threads_started":1}
I20260807 18:33:02.994539 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.972132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36369 (local address 127.0.0.1:40780)
0807 18:33:02.985632 (+ 13500us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:02.985650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:02.985770 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:02.986306 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:02.986310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:02.986336 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:33:02.986884 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:02.986900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.988893 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.988901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:02.992105 (+ 3204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:02.992120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.993995 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.993999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:02.994112 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.994132 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 18:33:02.994192 (+ 60us) client_negotiation.cc:240] Negotiation successful
0807 18:33:02.994264 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13370,"mutex_wait_us":1,"thread_start_us":13231,"threads_started":1}
I20260807 18:33:02.994755 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:02.972833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.27.1.193:36369)
0807 18:33:02.985222 (+ 12389us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:02.985231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:02.985763 (+ 532us) server_negotiation.cc:407] Connection header received
0807 18:33:02.985908 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:02.985916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:02.986112 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:02.986289 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:33:02.987160 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.988742 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.992336 (+ 3594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:02.993807 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:02.993941 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:02.994190 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:02.994264 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 18:33:02.994406 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12219,"thread_start_us":12063,"threads_started":1}
I20260807 18:33:02.995635 16927 tablet_service.cc:1467] Processing CreateTablet for tablet 69f57758f0ac4ede8517d5379308ef0d (DEFAULT_TABLE table=test-table [id=b1564f4cda7741faa090bf4b7e98c674]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:33:02.997283 16927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f57758f0ac4ede8517d5379308ef0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:02.997635 16791 tablet_service.cc:1467] Processing CreateTablet for tablet 69f57758f0ac4ede8517d5379308ef0d (DEFAULT_TABLE table=test-table [id=b1564f4cda7741faa090bf4b7e98c674]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:33:02.997895 16654 tablet_service.cc:1467] Processing CreateTablet for tablet 69f57758f0ac4ede8517d5379308ef0d (DEFAULT_TABLE table=test-table [id=b1564f4cda7741faa090bf4b7e98c674]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 18:33:02.998988 16791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f57758f0ac4ede8517d5379308ef0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:02.999228 16654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f57758f0ac4ede8517d5379308ef0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:03.010001 17015 tablet_bootstrap.cc:492] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a: Bootstrap starting.
I20260807 18:33:03.012138 17016 tablet_bootstrap.cc:492] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: Bootstrap starting.
I20260807 18:33:03.014398 17015 tablet_bootstrap.cc:654] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:03.015623 17016 tablet_bootstrap.cc:654] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:03.015846 17017 tablet_bootstrap.cc:492] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: Bootstrap starting.
I20260807 18:33:03.015873 17015 log.cc:824] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:03.017148 17016 log.cc:824] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:03.019222 17017 tablet_bootstrap.cc:654] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:03.019553 17015 tablet_bootstrap.cc:492] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a: No bootstrap required, opened a new log
I20260807 18:33:03.019728 17016 tablet_bootstrap.cc:492] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: No bootstrap required, opened a new log
I20260807 18:33:03.019838 17015 ts_tablet_manager.cc:1397] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 18:33:03.020012 17016 ts_tablet_manager.cc:1397] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 18:33:03.020390 17017 log.cc:824] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:03.024506 17017 tablet_bootstrap.cc:492] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: No bootstrap required, opened a new log
I20260807 18:33:03.024775 17017 ts_tablet_manager.cc:1397] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260807 18:33:03.027590 17016 raft_consensus.cc:348] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38d0c2b35c14639a063457d0c46ae51" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 43731 } } peers { permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 } } peers { permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 } }
I20260807 18:33:03.027938 17016 raft_consensus.cc:374] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:03.028074 17016 raft_consensus.cc:729] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38d0c2b35c14639a063457d0c46ae51, State: Initialized, Role: FOLLOWER
I20260807 18:33:03.028653 17016 consensus_queue.cc:260] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38d0c2b35c14639a063457d0c46ae51" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 43731 } } peers { permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 } } peers { permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 } }
I20260807 18:33:03.030329 17016 ts_tablet_manager.cc:1428] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260807 18:33:03.030308 17015 raft_consensus.cc:348] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38d0c2b35c14639a063457d0c46ae51" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 43731 } } peers { permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 } } peers { permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 } }
I20260807 18:33:03.030742 17015 raft_consensus.cc:374] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:03.030859 17015 raft_consensus.cc:729] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252b185b1a6f4d2798b3c1479168078a, State: Initialized, Role: FOLLOWER
I20260807 18:33:03.031010 16995 heartbeater.cc:503] Master 127.27.1.254:44647 was elected leader, sending a full tablet report...
I20260807 18:33:03.031514 17015 consensus_queue.cc:260] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38d0c2b35c14639a063457d0c46ae51" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 43731 } } peers { permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 } } peers { permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 } }
I20260807 18:33:03.033200 17015 ts_tablet_manager.cc:1428] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260807 18:33:03.036665 17017 raft_consensus.cc:348] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38d0c2b35c14639a063457d0c46ae51" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 43731 } } peers { permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 } } peers { permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 } }
I20260807 18:33:03.037043 17017 raft_consensus.cc:374] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:03.037146 17017 raft_consensus.cc:729] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e331f2c2c54792b70f5f1be2d81d44, State: Initialized, Role: FOLLOWER
I20260807 18:33:03.037722 17017 consensus_queue.cc:260] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38d0c2b35c14639a063457d0c46ae51" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 43731 } } peers { permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 } } peers { permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 } }
I20260807 18:33:03.038656 16859 heartbeater.cc:503] Master 127.27.1.254:44647 was elected leader, sending a full tablet report...
I20260807 18:33:03.039294 17017 ts_tablet_manager.cc:1428] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
W20260807 18:33:03.166747 16996 tablet.cc:2369] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:33:03.193287 16860 tablet.cc:2369] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:33:03.204850 16720 tablet.cc:2369] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:33:03.234627 17021 raft_consensus.cc:482] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:33:03.234968 17021 raft_consensus.cc:504] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38d0c2b35c14639a063457d0c46ae51" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 43731 } } peers { permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 } } peers { permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 } }
I20260807 18:33:03.236826 17021 leader_election.cc:288] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6e331f2c2c54792b70f5f1be2d81d44 (127.27.1.194:40145), 252b185b1a6f4d2798b3c1479168078a (127.27.1.193:36369)
I20260807 18:33:03.243258 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.237186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36369 (local address 127.27.1.195:60665)
0807 18:33:03.237399 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:03.237419 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:03.237556 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:03.237983 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:03.237988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:03.238024 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:33:03.238230 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:03.238241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.239570 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.239574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:03.241479 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:03.241509 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.242675 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.242679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:03.242788 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:03.242812 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 18:33:03.242881 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 18:33:03.242971 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 18:33:03.243355 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.237470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:60665 (local address 127.27.1.193:36369)
0807 18:33:03.237662 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:03.237666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:03.237682 (+ 16us) server_negotiation.cc:407] Connection header received
0807 18:33:03.237748 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:03.237751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:03.237815 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:03.237932 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:33:03.238404 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.239445 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.241638 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.242532 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.242692 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:03.242764 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:03.242931 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 18:33:03.243071 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 18:33:03.244300 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.237190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.1.195:56183 (local address 127.27.1.194:40145)
0807 18:33:03.237403 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:03.237411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:03.238463 (+ 1052us) server_negotiation.cc:407] Connection header received
0807 18:33:03.238581 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:03.238587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:03.238688 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:03.238784 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 18:33:03.239296 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.240274 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.242318 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.243466 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.243643 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:03.243710 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:03.244029 (+ 319us) server_negotiation.cc:299] Negotiation successful
0807 18:33:03.244122 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 18:33:03.244480 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.237187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.194:40145 (local address 127.27.1.195:56183)
0807 18:33:03.238355 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:03.238368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:03.238463 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:03.238803 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:03.238806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:03.238824 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 18:33:03.239118 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:03.239125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.240421 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.240428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:03.242138 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:03.242148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.243671 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.243677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:03.243848 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:03.243871 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 18:33:03.243992 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 18:33:03.244139 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":359,"thread_start_us":450,"threads_started":1}
I20260807 18:33:03.244495 16674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f57758f0ac4ede8517d5379308ef0d" candidate_uuid: "e38d0c2b35c14639a063457d0c46ae51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252b185b1a6f4d2798b3c1479168078a" is_pre_election: true
I20260807 18:33:03.245114 16674 raft_consensus.cc:2454] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38d0c2b35c14639a063457d0c46ae51 in term 0.
I20260807 18:33:03.245527 16812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f57758f0ac4ede8517d5379308ef0d" candidate_uuid: "e38d0c2b35c14639a063457d0c46ae51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" is_pre_election: true
I20260807 18:33:03.245833 16878 leader_election.cc:302] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252b185b1a6f4d2798b3c1479168078a, e38d0c2b35c14639a063457d0c46ae51; no voters:
I20260807 18:33:03.246038 16812 raft_consensus.cc:2454] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38d0c2b35c14639a063457d0c46ae51 in term 0.
I20260807 18:33:03.246241 17021 raft_consensus.cc:2781] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:33:03.246390 17021 raft_consensus.cc:482] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:33:03.246508 17021 raft_consensus.cc:3037] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:03.248831 17021 raft_consensus.cc:504] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38d0c2b35c14639a063457d0c46ae51" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 43731 } } peers { permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 } } peers { permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 } }
I20260807 18:33:03.249454 17021 leader_election.cc:288] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [CANDIDATE]: Term 1 election: Requested vote from peers b6e331f2c2c54792b70f5f1be2d81d44 (127.27.1.194:40145), 252b185b1a6f4d2798b3c1479168078a (127.27.1.193:36369)
I20260807 18:33:03.249884 16674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f57758f0ac4ede8517d5379308ef0d" candidate_uuid: "e38d0c2b35c14639a063457d0c46ae51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252b185b1a6f4d2798b3c1479168078a"
I20260807 18:33:03.249912 16812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f57758f0ac4ede8517d5379308ef0d" candidate_uuid: "e38d0c2b35c14639a063457d0c46ae51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e331f2c2c54792b70f5f1be2d81d44"
I20260807 18:33:03.250072 16674 raft_consensus.cc:3037] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:03.250111 16812 raft_consensus.cc:3037] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:03.252341 16674 raft_consensus.cc:2454] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38d0c2b35c14639a063457d0c46ae51 in term 1.
I20260807 18:33:03.252360 16812 raft_consensus.cc:2454] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38d0c2b35c14639a063457d0c46ae51 in term 1.
I20260807 18:33:03.252748 16878 leader_election.cc:302] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252b185b1a6f4d2798b3c1479168078a, e38d0c2b35c14639a063457d0c46ae51; no voters:
I20260807 18:33:03.253048 17021 raft_consensus.cc:2781] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:03.253459 17021 raft_consensus.cc:686] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 1 LEADER]: Becoming Leader. State: Replica: e38d0c2b35c14639a063457d0c46ae51, State: Running, Role: LEADER
I20260807 18:33:03.253913 17021 consensus_queue.cc:237] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38d0c2b35c14639a063457d0c46ae51" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 43731 } } peers { permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 } } peers { permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 } }
I20260807 18:33:03.258680 16525 catalog_manager.cc:5495] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 reported cstate change: term changed from 0 to 1, leader changed from <none> to e38d0c2b35c14639a063457d0c46ae51 (127.27.1.195). New cstate: current_term: 1 leader_uuid: "e38d0c2b35c14639a063457d0c46ae51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38d0c2b35c14639a063457d0c46ae51" member_type: VOTER last_known_addr { host: "127.27.1.195" port: 43731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 } health_report { overall_health: UNKNOWN } } }
I20260807 18:33:03.294055 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.289059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.27.1.195:43731)
0807 18:33:03.289273 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:03.289278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:03.289295 (+ 17us) server_negotiation.cc:407] Connection header received
0807 18:33:03.289441 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:03.289445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:03.289523 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:03.289647 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:03.289961 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.290692 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.292070 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.292461 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.292556 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:03.292960 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:03.293494 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:03.293615 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:03.293693 (+ 78us) server_negotiation.cc:299] Negotiation successful
0807 18:33:03.293755 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 18:33:03.301852 16812 raft_consensus.cc:1261] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44 [term 1 FOLLOWER]: Refusing update from remote peer e38d0c2b35c14639a063457d0c46ae51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:33:03.301853 16674 raft_consensus.cc:1261] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a [term 1 FOLLOWER]: Refusing update from remote peer e38d0c2b35c14639a063457d0c46ae51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:33:03.302801 17026 consensus_queue.cc:1035] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6e331f2c2c54792b70f5f1be2d81d44" member_type: VOTER last_known_addr { host: "127.27.1.194" port: 40145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:33:03.303164 17021 consensus_queue.cc:1035] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "252b185b1a6f4d2798b3c1479168078a" member_type: VOTER last_known_addr { host: "127.27.1.193" port: 36369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:33:03.320875 17030 mvcc.cc:205] Tried to move back new op lower bound from 7315978581195067392 to 7315978581011664896. Current Snapshot: MvccSnapshot[applied={T|T < 7315978581195067392}]
I20260807 18:33:03.322438 17029 mvcc.cc:205] Tried to move back new op lower bound from 7315978581195067392 to 7315978581011664896. Current Snapshot: MvccSnapshot[applied={T|T < 7315978581195067392}]
I20260807 18:33:03.337077 16525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:33:03.337343 16525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 18:33:03.341013 16525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91740e1bde64463484524c7139d22a93: Sending DeleteTablet for 3 replicas of tablet 69f57758f0ac4ede8517d5379308ef0d
I20260807 18:33:03.342105 16927 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f57758f0ac4ede8517d5379308ef0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:33:03 UTC) from {username='slave'} at 127.0.0.1:37120
I20260807 18:33:03.342329 16654 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f57758f0ac4ede8517d5379308ef0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:33:03 UTC) from {username='slave'} at 127.0.0.1:40780
I20260807 18:33:03.344472 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 16587
I20260807 18:33:03.345564 16791 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f57758f0ac4ede8517d5379308ef0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 18:33:03 UTC) from {username='slave'} at 127.0.0.1:39650
I20260807 18:33:03.352838 17041 tablet_replica.cc:330] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a: stopping tablet replica
I20260807 18:33:03.353410 17041 raft_consensus.cc:2229] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:33:03.354074 17041 raft_consensus.cc:2258] T 69f57758f0ac4ede8517d5379308ef0d P 252b185b1a6f4d2798b3c1479168078a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:33:03.362106 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.346237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40784 (local address 127.27.1.193:36369)
0807 18:33:03.346756 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:03.346766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:03.351223 (+ 4457us) server_negotiation.cc:407] Connection header received
0807 18:33:03.352213 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:03.352219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:03.352355 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:03.352439 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:03.353846 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.355377 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.357292 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.358158 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.358286 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:03.360259 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:03.360274 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:03.360294 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:03.361031 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:03.361086 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:03.361112 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:03.361217 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:03.361544 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 18:33:03.361694 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
I20260807 18:33:03.363261 16714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:03.363356 17040 tablet_replica.cc:330] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: stopping tablet replica
I20260807 18:33:03.363955 17040 raft_consensus.cc:2229] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 1 LEADER]: Raft consensus shutting down.
I20260807 18:33:03.364874 17040 raft_consensus.cc:2258] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:33:03.367537 17040 ts_tablet_manager.cc:1905] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:33:03.376840 17040 ts_tablet_manager.cc:1918] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:33:03.377048 17040 log.cc:1196] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-2/wal/wals/69f57758f0ac4ede8517d5379308ef0d
I20260807 18:33:03.377704 17040 ts_tablet_manager.cc:1939] T 69f57758f0ac4ede8517d5379308ef0d P e38d0c2b35c14639a063457d0c46ae51: Deleting consensus metadata
I20260807 18:33:03.378367 17043 tablet_replica.cc:330] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: stopping tablet replica
I20260807 18:33:03.378904 17043 raft_consensus.cc:2229] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 18:33:03.379513 17043 raft_consensus.cc:2258] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 18:33:03.380235 16510 catalog_manager.cc:4841] TS e38d0c2b35c14639a063457d0c46ae51 (127.27.1.195:43731): tablet 69f57758f0ac4ede8517d5379308ef0d (table test-table [id=b1564f4cda7741faa090bf4b7e98c674]) successfully deleted
I20260807 18:33:03.381956 17043 ts_tablet_manager.cc:1905] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 18:33:03.388119 17043 ts_tablet_manager.cc:1918] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 18:33:03.388291 17043 log.cc:1196] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: Deleting WAL directory at /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786127255008662-27655-0/minicluster-data/ts-1/wal/wals/69f57758f0ac4ede8517d5379308ef0d
I20260807 18:33:03.388759 17043 ts_tablet_manager.cc:1939] T 69f57758f0ac4ede8517d5379308ef0d P b6e331f2c2c54792b70f5f1be2d81d44: Deleting consensus metadata
I20260807 18:33:03.390347 16512 catalog_manager.cc:4841] TS b6e331f2c2c54792b70f5f1be2d81d44 (127.27.1.194:40145): tablet 69f57758f0ac4ede8517d5379308ef0d (table test-table [id=b1564f4cda7741faa090bf4b7e98c674]) successfully deleted
I20260807 18:33:03.480420 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 16587
I20260807 18:33:03.506224 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 16723
W20260807 18:33:03.506266 16511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.1.193:36369 (error 108)
I20260807 18:33:03.509886 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.509197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36369 (local address 127.0.0.1:40786)
0807 18:33:03.509621 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:03.509731 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:36369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":222,"threads_started":1}
W20260807 18:33:03.510206 16511 catalog_manager.cc:4584] TS 252b185b1a6f4d2798b3c1479168078a (127.27.1.193:36369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69f57758f0ac4ede8517d5379308ef0d: Network error: Client connection negotiation failed: client connection to 127.27.1.193:36369: connect: Connection refused (error 111)
I20260807 18:33:03.514971 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.507422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.27.1.194:40145)
0807 18:33:03.507688 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:03.507697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:03.507722 (+ 25us) server_negotiation.cc:407] Connection header received
0807 18:33:03.507813 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:03.507821 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:03.507962 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:03.508097 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:03.508648 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.509876 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.511808 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.512239 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.512337 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:03.513550 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:03.513562 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:03.513575 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:03.514179 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:03.514223 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:03.514242 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:03.514327 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:03.514501 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 18:33:03.514621 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260807 18:33:03.515681 16854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:03.563907 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.563334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36369 (local address 127.0.0.1:40788)
0807 18:33:03.563583 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:03.563698 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:36369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 18:33:03.565161 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.564709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36369 (local address 127.0.0.1:40802)
0807 18:33:03.564900 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:03.564993 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:36369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 18:33:03.606976 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 16723
I20260807 18:33:03.626192 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.625576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36369 (local address 127.0.0.1:40814)
0807 18:33:03.625854 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:03.625977 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:36369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260807 18:33:03.627579 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.627090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36369 (local address 127.0.0.1:40822)
0807 18:33:03.627297 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:03.627406 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:36369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 18:33:03.628216 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 16862
I20260807 18:33:03.636363 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.628997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.27.1.195:43731)
0807 18:33:03.629186 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:03.629192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:03.629215 (+ 23us) server_negotiation.cc:407] Connection header received
0807 18:33:03.629374 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:03.629379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:03.629491 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:03.629611 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:03.630029 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.631274 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.632577 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:03.633243 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:03.633385 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:03.634430 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:03.634443 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:03.634472 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:03.635503 (+ 1031us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:03.635572 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:03.635604 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:03.635725 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:03.635852 (+ 127us) server_negotiation.cc:299] Negotiation successful
0807 18:33:03.635950 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 18:33:03.637140 16990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:03.708292 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.707797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36369 (local address 127.0.0.1:40836)
0807 18:33:03.708040 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:03.708131 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:36369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260807 18:33:03.709460 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.709032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36369 (local address 127.0.0.1:40842)
0807 18:33:03.709261 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:03.709343 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:36369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260807 18:33:03.757225 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 16862
I20260807 18:33:03.780175 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 16494
I20260807 18:33:03.781301 16558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:03.894735 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:03.893923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.193:36369 (local address 127.0.0.1:40858)
0807 18:33:03.894138 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:03.894440 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.193:36369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 18:33:03.894990 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 16494
2026-08-07T18:33:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3132 ms)
[----------] 6 tests from EncryptionPolicyTest (23301 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:04 dist-test-slave-1xrh krb5kdc[17052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:04 dist-test-slave-1xrh krb5kdc[17052](info): set up 2 sockets
Aug 07 18:33:04 dist-test-slave-1xrh krb5kdc[17052](info): commencing operation
krb5kdc: starting...
W20260807 18:33:05.072724 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:05 dist-test-slave-1xrh krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:05.372797 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:05.715894 17066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:05.716167 17066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:05.716336 17066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:05.723760 17066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:05.723896 17066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:05.724002 17066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:05.724107 17066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:05.724169 17066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:05.734042 17066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:05.735376 17066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:05.736688 17066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:05.748713 17071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:05.748662 17075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:05.748687 17073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:05.749529 17066 server_base.cc:1034] running on GCE node
I20260807 18:33:05.750301 17066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:05.751817 17066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:05.753055 17066 hybrid_clock.cc:648] HybridClock initialized: now 1786127585753015 us; error 34 us; skew 500 ppm
I20260807 18:33:05.753423 17066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:05 dist-test-slave-1xrh krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:05.756561 17066 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:05.758023 17066 webserver.cc:460] Webserver started at http://127.0.0.1:39437/ using document root <none> and password file <none>
I20260807 18:33:05.758687 17066 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:05.758834 17066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:05.759191 17066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:05.762220 17066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "56555b6ff8cd4294aab9e1b35b3ba54b"
format_stamp: "Formatted at 2026-08-07 18:33:05 on dist-test-slave-1xrh"
server_key: "4cff46ab928bb9e6e19bef4c191da8f5"
server_key_iv: "c2541bc1f0ed47c1bd0143157fd54ae6"
server_key_version: "encryptionkey@0"
I20260807 18:33:05.763105 17066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "56555b6ff8cd4294aab9e1b35b3ba54b"
format_stamp: "Formatted at 2026-08-07 18:33:05 on dist-test-slave-1xrh"
server_key: "4cff46ab928bb9e6e19bef4c191da8f5"
server_key_iv: "c2541bc1f0ed47c1bd0143157fd54ae6"
server_key_version: "encryptionkey@0"
I20260807 18:33:05.768244 17066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 18:33:05.771780 17084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:05.773049 17066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:33:05.773232 17066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "56555b6ff8cd4294aab9e1b35b3ba54b"
format_stamp: "Formatted at 2026-08-07 18:33:05 on dist-test-slave-1xrh"
server_key: "4cff46ab928bb9e6e19bef4c191da8f5"
server_key_iv: "c2541bc1f0ed47c1bd0143157fd54ae6"
server_key_version: "encryptionkey@0"
I20260807 18:33:05.773447 17066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:05.786736 17066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:05.790611 17066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:05.790910 17066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:05.809563 17066 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41653
I20260807 18:33:05.809587 17137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41653 every 8 connection(s)
I20260807 18:33:05.810940 17066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:05.812402 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 17066
I20260807 18:33:05.812633 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:05.813513 27655 external_mini_cluster.cc:1428] Setting key 66d56c81b8a193cccbb1c566333782df
I20260807 18:33:05.815775 17138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 18:33:05 dist-test-slave-1xrh krb5kdc[17052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 18:33:05.829386 17138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b: Bootstrap starting.
I20260807 18:33:05.833786 17138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:05.835117 17138 log.cc:824] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:05.838410 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:05.815507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57649 (local address 127.0.0.1:41653)
0807 18:33:05.816364 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:05.816375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:05.816415 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:33:05.818017 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:05.818037 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:05.818390 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:05.818715 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:05.820668 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:05.821998 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:05.823825 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:05.824450 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:05.824600 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:05.829423 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:05.829436 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:05.829455 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:05.829503 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:05.833488 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:05.834411 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:05.834420 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:05.834428 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:05.834636 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:05.835110 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:05.835116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:05.835140 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:05.835867 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:05.836166 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:05.836757 (+ 591us) server_negotiation.cc:299] Negotiation successful
0807 18:33:05.837109 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":457,"threads_started":1}
I20260807 18:33:05.838673 17138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b: No bootstrap required, opened a new log
I20260807 18:33:05.842382 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:05.846937 17138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56555b6ff8cd4294aab9e1b35b3ba54b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260807 18:33:05.847457 17138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:05.847677 17138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56555b6ff8cd4294aab9e1b35b3ba54b, State: Initialized, Role: FOLLOWER
I20260807 18:33:05.848413 17138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56555b6ff8cd4294aab9e1b35b3ba54b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260807 18:33:05.848739 17138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:05.848896 17138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:05.849081 17138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:05.852176 17138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56555b6ff8cd4294aab9e1b35b3ba54b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260807 18:33:05.852759 17138 leader_election.cc:302] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56555b6ff8cd4294aab9e1b35b3ba54b; no voters:
I20260807 18:33:05.853416 17138 leader_election.cc:288] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:05.853587 17148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:05.857507 17148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [term 1 LEADER]: Becoming Leader. State: Replica: 56555b6ff8cd4294aab9e1b35b3ba54b, State: Running, Role: LEADER
I20260807 18:33:05.858217 17148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56555b6ff8cd4294aab9e1b35b3ba54b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260807 18:33:05.859871 17138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:05.861508 17149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56555b6ff8cd4294aab9e1b35b3ba54b. Latest consensus state: current_term: 1 leader_uuid: "56555b6ff8cd4294aab9e1b35b3ba54b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56555b6ff8cd4294aab9e1b35b3ba54b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } } }
I20260807 18:33:05.861485 17150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56555b6ff8cd4294aab9e1b35b3ba54b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56555b6ff8cd4294aab9e1b35b3ba54b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } } }
I20260807 18:33:05.861940 17149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:05.861940 17150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:05.867731 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:05.847342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38661 (local address 127.0.0.1:41653)
0807 18:33:05.847582 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:05.847591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:05.848044 (+ 453us) server_negotiation.cc:407] Connection header received
0807 18:33:05.848229 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:05.848233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:05.848291 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:05.848370 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:05.850334 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:05.851114 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:05.853065 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:05.853880 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:05.854027 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:05.858043 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:05.858054 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:05.858058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:05.858108 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:05.862161 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:05.863644 (+ 1483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:05.863652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:05.863658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:05.863887 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:05.864638 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:05.864646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:05.864651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:05.865148 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:05.866703 (+ 1555us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:05.866928 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 18:33:05.867196 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:33:05.877357 17153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:05.883071 17153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:05.893234 17153 catalog_manager.cc:1347] Generated new cluster ID: 634c250812c0472198df15d72e76bb01
I20260807 18:33:05.893347 17153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:05.899946 17153 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:05.901701 17153 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:05.909646 17153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56555b6ff8cd4294aab9e1b35b3ba54b: Generated new TSK 0
I20260807 18:33:05.910429 17153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:05.922539 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17066
I20260807 18:33:05.923828 17133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:06.013901 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 17066
W20260807 18:33:06.034950 17050 connection.cc:535] client connection to 127.0.0.1:41653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41653): Connection reset by peer (error 104)
2026-08-07T18:33:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2107 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:06 dist-test-slave-1xrh krb5kdc[17172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:06 dist-test-slave-1xrh krb5kdc[17172](info): set up 2 sockets
Aug 07 18:33:06 dist-test-slave-1xrh krb5kdc[17172](info): commencing operation
krb5kdc: starting...
W20260807 18:33:07.173064 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:07 dist-test-slave-1xrh krb5kdc[17172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:07.489974 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:07.829528 17185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:07.829873 17185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:07.830039 17185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:07.837173 17185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:07.837320 17185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:07.837428 17185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:07.837543 17185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:07.837626 17185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:07.847564 17185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:07.848939 17185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:07.850277 17185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:07.862280 17191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:07.862314 17192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:07.862860 17194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:07.863294 17185 server_base.cc:1034] running on GCE node
I20260807 18:33:07.864056 17185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:07.865553 17185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:07.866784 17185 hybrid_clock.cc:648] HybridClock initialized: now 1786127587866728 us; error 45 us; skew 500 ppm
I20260807 18:33:07.867168 17185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:07 dist-test-slave-1xrh krb5kdc[17172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:07.874717 17185 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:07.876351 17185 webserver.cc:460] Webserver started at http://127.0.0.1:34265/ using document root <none> and password file <none>
I20260807 18:33:07.877039 17185 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:07.877214 17185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:07.877549 17185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:07.880637 17185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "e58968b5c2b8426db5b1dc1e081ff2b9"
format_stamp: "Formatted at 2026-08-07 18:33:07 on dist-test-slave-1xrh"
server_key: "e81d2e3bb7b7eaa0d55a392a64c5b5e9"
server_key_iv: "d4c3a27b9858959107150487303e1483"
server_key_version: "encryptionkey@0"
I20260807 18:33:07.881436 17185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "e58968b5c2b8426db5b1dc1e081ff2b9"
format_stamp: "Formatted at 2026-08-07 18:33:07 on dist-test-slave-1xrh"
server_key: "e81d2e3bb7b7eaa0d55a392a64c5b5e9"
server_key_iv: "d4c3a27b9858959107150487303e1483"
server_key_version: "encryptionkey@0"
I20260807 18:33:07.886574 17185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 18:33:07.890085 17201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:07.891454 17185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 18:33:07.891647 17185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "e58968b5c2b8426db5b1dc1e081ff2b9"
format_stamp: "Formatted at 2026-08-07 18:33:07 on dist-test-slave-1xrh"
server_key: "e81d2e3bb7b7eaa0d55a392a64c5b5e9"
server_key_iv: "d4c3a27b9858959107150487303e1483"
server_key_version: "encryptionkey@0"
I20260807 18:33:07.891852 17185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:07.924129 17185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:07.928248 17185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:07.928557 17185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:07.947368 17185 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39759
I20260807 18:33:07.947412 17255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39759 every 8 connection(s)
I20260807 18:33:07.948751 17185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:07.949100 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 17185
I20260807 18:33:07.949285 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:07.949883 27655 external_mini_cluster.cc:1428] Setting key c23704119d9dc08aff7013004eef9fc3
I20260807 18:33:07.953631 17256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:07.965400 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:07.951817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37249 (local address 127.0.0.1:39759)
0807 18:33:07.952646 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:07.952661 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:07.952712 (+ 51us) server_negotiation.cc:407] Connection header received
0807 18:33:07.955023 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:07.955050 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:07.955553 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:07.956022 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:07.956526 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:07.958566 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:07.960033 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:07.961048 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:07.961226 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:07.962451 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:07.962462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:07.962480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:07.963268 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:07.963357 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:07.963396 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:07.963585 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:07.963854 (+ 269us) server_negotiation.cc:299] Negotiation successful
0807 18:33:07.964043 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":328,"threads_started":1}
I20260807 18:33:07.967680 17256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9: Bootstrap starting.
I20260807 18:33:07.969610 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:07.971225 17256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:07.972222 17256 log.cc:824] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:07.975485 17256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9: No bootstrap required, opened a new log
I20260807 18:33:07.981019 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:07.974152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50281 (local address 127.0.0.1:39759)
0807 18:33:07.974327 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:07.974333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:07.974670 (+ 337us) server_negotiation.cc:407] Connection header received
0807 18:33:07.974807 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:07.974812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:07.974891 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:07.975018 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:07.975578 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:07.976755 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:07.978132 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:07.978856 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:07.978995 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:07.980194 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:07.980206 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:07.980211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:07.980232 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:07.980283 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:07.980301 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:07.980430 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:07.980588 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 18:33:07.980649 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 18:33:07.985456 17256 raft_consensus.cc:348] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58968b5c2b8426db5b1dc1e081ff2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } }
I20260807 18:33:07.985831 17256 raft_consensus.cc:374] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:07.985914 17256 raft_consensus.cc:729] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58968b5c2b8426db5b1dc1e081ff2b9, State: Initialized, Role: FOLLOWER
I20260807 18:33:07.986498 17256 consensus_queue.cc:260] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58968b5c2b8426db5b1dc1e081ff2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } }
I20260807 18:33:07.986712 17256 raft_consensus.cc:388] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:07.986814 17256 raft_consensus.cc:482] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:07.986953 17256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:07.989399 17256 raft_consensus.cc:504] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58968b5c2b8426db5b1dc1e081ff2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } }
I20260807 18:33:07.989921 17256 leader_election.cc:302] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e58968b5c2b8426db5b1dc1e081ff2b9; no voters:
I20260807 18:33:07.990490 17256 leader_election.cc:288] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:07.990651 17266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:07.991466 17266 raft_consensus.cc:686] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [term 1 LEADER]: Becoming Leader. State: Replica: e58968b5c2b8426db5b1dc1e081ff2b9, State: Running, Role: LEADER
I20260807 18:33:07.992120 17266 consensus_queue.cc:237] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58968b5c2b8426db5b1dc1e081ff2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } }
I20260807 18:33:07.992748 17256 sys_catalog.cc:564] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:07.995034 17268 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e58968b5c2b8426db5b1dc1e081ff2b9. Latest consensus state: current_term: 1 leader_uuid: "e58968b5c2b8426db5b1dc1e081ff2b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58968b5c2b8426db5b1dc1e081ff2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } } }
I20260807 18:33:07.995461 17268 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:07.996171 17267 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e58968b5c2b8426db5b1dc1e081ff2b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58968b5c2b8426db5b1dc1e081ff2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } } }
I20260807 18:33:07.996513 17267 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:07.996783 17273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:08.008792 17273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:08.019826 17273 catalog_manager.cc:1347] Generated new cluster ID: b92dad5b04cf460ca5edad04e988e332
I20260807 18:33:08.019949 17273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:08.031057 17273 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:08.032335 17273 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:08.037768 17273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e58968b5c2b8426db5b1dc1e081ff2b9: Generated new TSK 0
I20260807 18:33:08.038554 17273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:08.049551 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17185
W20260807 18:33:08.050686 17251 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37249
W20260807 18:33:08.051713 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17185: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:08.051973 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 17185
W20260807 18:33:08.074071 17170 connection.cc:535] client connection to 127.0.0.1:39759 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39759): Connection reset by peer (error 104)
2026-08-07T18:33:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:08 dist-test-slave-1xrh krb5kdc[17288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:08 dist-test-slave-1xrh krb5kdc[17288](info): set up 2 sockets
Aug 07 18:33:08 dist-test-slave-1xrh krb5kdc[17288](info): commencing operation
krb5kdc: starting...
W20260807 18:33:09.221129 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:09 dist-test-slave-1xrh krb5kdc[17288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:09.542959 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:09.885522 17301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:09.885793 17301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:09.885883 17301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:09.892992 17301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:09.893100 17301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:09.893147 17301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:09.893213 17301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:09.893254 17301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:09.903882 17301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:09.905424 17301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:09.906734 17301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:09.919214 17306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:09.919243 17309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:09.919288 17307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:09.920486 17301 server_base.cc:1034] running on GCE node
I20260807 18:33:09.921303 17301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:09.922837 17301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:09.924080 17301 hybrid_clock.cc:648] HybridClock initialized: now 1786127589924013 us; error 51 us; skew 500 ppm
I20260807 18:33:09.924474 17301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:09 dist-test-slave-1xrh krb5kdc[17288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:09.927858 17301 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:09.929327 17301 webserver.cc:460] Webserver started at http://127.0.0.1:37639/ using document root <none> and password file <none>
I20260807 18:33:09.930063 17301 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:09.930230 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:09.930583 17301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:09.933960 17301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "f228deb0afb74cc0855383d50d8a0d80"
format_stamp: "Formatted at 2026-08-07 18:33:09 on dist-test-slave-1xrh"
server_key: "7075b14fa2d6834b165d5caf30330bab"
server_key_iv: "b4277bf880e613c5d46ff6a74e67549e"
server_key_version: "encryptionkey@0"
I20260807 18:33:09.934818 17301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "f228deb0afb74cc0855383d50d8a0d80"
format_stamp: "Formatted at 2026-08-07 18:33:09 on dist-test-slave-1xrh"
server_key: "7075b14fa2d6834b165d5caf30330bab"
server_key_iv: "b4277bf880e613c5d46ff6a74e67549e"
server_key_version: "encryptionkey@0"
I20260807 18:33:09.940320 17301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260807 18:33:09.943948 17316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:09.945294 17301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:33:09.945487 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "f228deb0afb74cc0855383d50d8a0d80"
format_stamp: "Formatted at 2026-08-07 18:33:09 on dist-test-slave-1xrh"
server_key: "7075b14fa2d6834b165d5caf30330bab"
server_key_iv: "b4277bf880e613c5d46ff6a74e67549e"
server_key_version: "encryptionkey@0"
I20260807 18:33:09.945698 17301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:09.952034 17301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:09.956305 17301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:09.956648 17301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:09.977029 17301 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41513
I20260807 18:33:09.977077 17370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41513 every 8 connection(s)
I20260807 18:33:09.978610 17301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:09.982547 17371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:09.984081 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 17301
I20260807 18:33:09.984297 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:09.984819 27655 external_mini_cluster.cc:1428] Setting key 5a5f9b6588fca9613c7776851a192181
I20260807 18:33:09.993861 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:09.986895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55131 (local address 127.0.0.1:41513)
0807 18:33:09.987859 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:09.987875 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:09.987933 (+ 58us) server_negotiation.cc:407] Connection header received
0807 18:33:09.989707 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:09.989735 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:09.990220 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:09.990692 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:09.991777 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:09.991785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:09.991802 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:09.991820 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:09.991895 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:09.991932 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:09.992158 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:09.992291 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 18:33:09.992512 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":445,"threads_started":1}
I20260807 18:33:09.997447 17371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80: Bootstrap starting.
I20260807 18:33:09.997988 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:10.001412 17371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:10.002956 17371 log.cc:824] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:10.006559 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:10.004097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60405 (local address 127.0.0.1:41513)
0807 18:33:10.004305 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:10.004314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:10.004586 (+ 272us) server_negotiation.cc:407] Connection header received
0807 18:33:10.004784 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:10.004791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:10.004859 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:10.004988 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:10.005908 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:10.005914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:10.005918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:10.005936 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:10.005991 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:10.006010 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:10.006134 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:10.006207 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 18:33:10.006270 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 18:33:10.006785 17371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80: No bootstrap required, opened a new log
I20260807 18:33:10.014974 17371 raft_consensus.cc:348] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f228deb0afb74cc0855383d50d8a0d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } }
I20260807 18:33:10.015455 17371 raft_consensus.cc:374] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:10.015575 17371 raft_consensus.cc:729] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f228deb0afb74cc0855383d50d8a0d80, State: Initialized, Role: FOLLOWER
I20260807 18:33:10.016218 17371 consensus_queue.cc:260] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f228deb0afb74cc0855383d50d8a0d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } }
I20260807 18:33:10.016466 17371 raft_consensus.cc:388] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:10.016626 17371 raft_consensus.cc:482] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:10.016798 17371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:10.019105 17371 raft_consensus.cc:504] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f228deb0afb74cc0855383d50d8a0d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } }
I20260807 18:33:10.019676 17371 leader_election.cc:302] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f228deb0afb74cc0855383d50d8a0d80; no voters:
I20260807 18:33:10.020402 17371 leader_election.cc:288] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:10.020534 17381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:10.021248 17381 raft_consensus.cc:686] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [term 1 LEADER]: Becoming Leader. State: Replica: f228deb0afb74cc0855383d50d8a0d80, State: Running, Role: LEADER
I20260807 18:33:10.021864 17381 consensus_queue.cc:237] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f228deb0afb74cc0855383d50d8a0d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } }
I20260807 18:33:10.022568 17371 sys_catalog.cc:564] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:10.025130 17383 sys_catalog.cc:455] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f228deb0afb74cc0855383d50d8a0d80. Latest consensus state: current_term: 1 leader_uuid: "f228deb0afb74cc0855383d50d8a0d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f228deb0afb74cc0855383d50d8a0d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } } }
I20260807 18:33:10.025019 17382 sys_catalog.cc:455] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f228deb0afb74cc0855383d50d8a0d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f228deb0afb74cc0855383d50d8a0d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } } }
I20260807 18:33:10.025632 17382 sys_catalog.cc:458] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:10.025692 17383 sys_catalog.cc:458] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:10.032439 17391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:10.036806 17391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:10.044834 17391 catalog_manager.cc:1347] Generated new cluster ID: 00d7890c5ac84fe9a875d3b7015b8a30
I20260807 18:33:10.044939 17391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:10.058243 17391 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:10.058756 17391 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:10.065363 17391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f228deb0afb74cc0855383d50d8a0d80: Generated new TSK 0
I20260807 18:33:10.066082 17391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:10.075856 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17301
W20260807 18:33:10.076939 17366 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55131
W20260807 18:33:10.077625 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17301: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:10.077781 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 17301
2026-08-07T18:33:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:10 dist-test-slave-1xrh krb5kdc[17404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:10 dist-test-slave-1xrh krb5kdc[17404](info): set up 2 sockets
Aug 07 18:33:10 dist-test-slave-1xrh krb5kdc[17404](info): commencing operation
krb5kdc: starting...
W20260807 18:33:11.239514 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:11 dist-test-slave-1xrh krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:11.558108 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:11.899482 17417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:11.899740 17417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:11.899917 17417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:11.906965 17417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:11.907059 17417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:11.907182 17417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:11.907305 17417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:11.907366 17417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:11.918104 17417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:11.919437 17417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:11.920709 17417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:11.932725 17422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:11.932646 17425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:11.932699 17423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:11.933782 17417 server_base.cc:1034] running on GCE node
I20260807 18:33:11.934543 17417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:11.935992 17417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:11.937229 17417 hybrid_clock.cc:648] HybridClock initialized: now 1786127591937182 us; error 37 us; skew 500 ppm
I20260807 18:33:11.937594 17417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:11 dist-test-slave-1xrh krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:11.940824 17417 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:11.942318 17417 webserver.cc:460] Webserver started at http://127.0.0.1:34595/ using document root <none> and password file <none>
I20260807 18:33:11.942992 17417 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:11.943194 17417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:11.943509 17417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:11.946427 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "1566c884945c4831ac033872465ba1f2"
format_stamp: "Formatted at 2026-08-07 18:33:11 on dist-test-slave-1xrh"
server_key: "bc64dfd929024f1b55b679c210b828c8"
server_key_iv: "5d2fb00f9ae3e9ab140c3c3a3d6f2408"
server_key_version: "encryptionkey@0"
I20260807 18:33:11.947300 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "1566c884945c4831ac033872465ba1f2"
format_stamp: "Formatted at 2026-08-07 18:33:11 on dist-test-slave-1xrh"
server_key: "bc64dfd929024f1b55b679c210b828c8"
server_key_iv: "5d2fb00f9ae3e9ab140c3c3a3d6f2408"
server_key_version: "encryptionkey@0"
I20260807 18:33:11.952312 17417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 18:33:11.955703 17432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:11.957001 17417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:33:11.957211 17417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "1566c884945c4831ac033872465ba1f2"
format_stamp: "Formatted at 2026-08-07 18:33:11 on dist-test-slave-1xrh"
server_key: "bc64dfd929024f1b55b679c210b828c8"
server_key_iv: "5d2fb00f9ae3e9ab140c3c3a3d6f2408"
server_key_version: "encryptionkey@0"
I20260807 18:33:11.957406 17417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:11.981011 17417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:11.985487 17417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:11.985790 17417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:12.004885 17417 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42741
I20260807 18:33:12.004923 17486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42741 every 8 connection(s)
I20260807 18:33:12.006325 17417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:12.010154 17487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:12.016582 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 17417
I20260807 18:33:12.016790 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:12.017427 27655 external_mini_cluster.cc:1428] Setting key 964ef5f3032865317f9c53e83a9202e2
I20260807 18:33:12.021375 17487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2: Bootstrap starting.
I20260807 18:33:12.024456 17487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:12.025516 17487 log.cc:824] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:12.028216 17487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2: No bootstrap required, opened a new log
Aug 07 18:33:12 dist-test-slave-1xrh krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 18:33:12.036453 17487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1566c884945c4831ac033872465ba1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } }
I20260807 18:33:12.036823 17487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:12.036939 17487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1566c884945c4831ac033872465ba1f2, State: Initialized, Role: FOLLOWER
I20260807 18:33:12.037663 17487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1566c884945c4831ac033872465ba1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } }
I20260807 18:33:12.038031 17487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:12.038203 17487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:12.038408 17487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:12.040248 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:12.019391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59311 (local address 127.0.0.1:42741)
0807 18:33:12.020176 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:12.020188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:12.020229 (+ 41us) server_negotiation.cc:407] Connection header received
0807 18:33:12.021957 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:12.021980 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:12.022343 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:12.022741 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:12.024295 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:12.025728 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:12.026952 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:12.027685 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:12.027896 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:12.032123 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:12.032130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:12.032146 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:12.032184 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:12.035507 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:12.036233 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:12.036240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:12.036247 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:12.036411 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:12.036878 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:12.036885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:12.036890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:12.037632 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:12.038022 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:12.038371 (+ 349us) server_negotiation.cc:299] Negotiation successful
0807 18:33:12.038829 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":340,"threads_started":1}
I20260807 18:33:12.041430 17487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1566c884945c4831ac033872465ba1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } }
I20260807 18:33:12.041935 17487 leader_election.cc:302] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1566c884945c4831ac033872465ba1f2; no voters:
I20260807 18:33:12.042568 17487 leader_election.cc:288] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:12.042689 17492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:12.043955 17492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [term 1 LEADER]: Becoming Leader. State: Replica: 1566c884945c4831ac033872465ba1f2, State: Running, Role: LEADER
I20260807 18:33:12.044265 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:12.044584 17492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1566c884945c4831ac033872465ba1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } }
I20260807 18:33:12.044865 17487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:12.052870 17494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1566c884945c4831ac033872465ba1f2. Latest consensus state: current_term: 1 leader_uuid: "1566c884945c4831ac033872465ba1f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1566c884945c4831ac033872465ba1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } } }
I20260807 18:33:12.053356 17494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:12.054062 17493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1566c884945c4831ac033872465ba1f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1566c884945c4831ac033872465ba1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } } }
I20260807 18:33:12.054461 17493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:12.075140 17501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:12.082791 17501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:12.096138 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:12.067731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57033 (local address 127.0.0.1:42741)
0807 18:33:12.067951 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:12.067960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:12.078321 (+ 10361us) server_negotiation.cc:407] Connection header received
0807 18:33:12.078669 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:12.078675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:12.078753 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:12.078826 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:12.081086 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:12.082215 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:12.084163 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:12.084885 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:12.084993 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:12.088542 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:12.088552 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:12.088589 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:12.088631 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:12.092769 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:12.093664 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:12.093670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:12.093674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:12.093812 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:12.094297 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:12.094302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:12.094305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:12.094731 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:12.094943 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:12.095558 (+ 615us) server_negotiation.cc:299] Negotiation successful
0807 18:33:12.095773 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:33:12.101517 17501 catalog_manager.cc:1347] Generated new cluster ID: bc345bedb0654ee885288344b3ff8712
I20260807 18:33:12.101635 17501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:12.128238 17501 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:12.129606 17501 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:12.140956 17501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1566c884945c4831ac033872465ba1f2: Generated new TSK 0
I20260807 18:33:12.141768 17501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:12.158044 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17417
I20260807 18:33:12.159479 17482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:12.252246 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 17417
2026-08-07T18:33:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2174 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:12 dist-test-slave-1xrh krb5kdc[17520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:12 dist-test-slave-1xrh krb5kdc[17520](info): set up 2 sockets
Aug 07 18:33:12 dist-test-slave-1xrh krb5kdc[17520](info): commencing operation
krb5kdc: starting...
W20260807 18:33:13.452790 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:13 dist-test-slave-1xrh krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:13.778221 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:14.119771 17535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:14.120114 17535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:14.120275 17535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:14.127480 17535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:14.127622 17535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:14.127736 17535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:14.127846 17535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:14.127936 17535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:14.137720 17535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:14.139135 17535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:14.140480 17535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:14.152473 17544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:14.152565 17540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:14.152587 17542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:14.153158 17535 server_base.cc:1034] running on GCE node
I20260807 18:33:14.154292 17535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:14.155972 17535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:14.157202 17535 hybrid_clock.cc:648] HybridClock initialized: now 1786127594157139 us; error 51 us; skew 500 ppm
I20260807 18:33:14.157577 17535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:14 dist-test-slave-1xrh krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:14.168036 17535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:14.169567 17535 webserver.cc:460] Webserver started at http://127.0.0.1:33057/ using document root <none> and password file <none>
I20260807 18:33:14.170286 17535 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:14.170425 17535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:14.170745 17535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:14.173834 17535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "70cabc4895284c38a95490e7a80a0395"
format_stamp: "Formatted at 2026-08-07 18:33:14 on dist-test-slave-1xrh"
server_key: "2c833ec3a53f64cfa269979442aeaad1"
server_key_iv: "681462e2e684ff5301084d1bdafd56c7"
server_key_version: "encryptionkey@0"
I20260807 18:33:14.174660 17535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "70cabc4895284c38a95490e7a80a0395"
format_stamp: "Formatted at 2026-08-07 18:33:14 on dist-test-slave-1xrh"
server_key: "2c833ec3a53f64cfa269979442aeaad1"
server_key_iv: "681462e2e684ff5301084d1bdafd56c7"
server_key_version: "encryptionkey@0"
I20260807 18:33:14.180006 17535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 18:33:14.183658 17551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:14.185006 17535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 18:33:14.185220 17535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "70cabc4895284c38a95490e7a80a0395"
format_stamp: "Formatted at 2026-08-07 18:33:14 on dist-test-slave-1xrh"
server_key: "2c833ec3a53f64cfa269979442aeaad1"
server_key_iv: "681462e2e684ff5301084d1bdafd56c7"
server_key_version: "encryptionkey@0"
I20260807 18:33:14.185428 17535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:14.200205 17535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:14.204123 17535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:14.204403 17535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:14.223344 17535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33673
I20260807 18:33:14.223382 17602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33673 every 8 connection(s)
I20260807 18:33:14.224689 17535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:14.228253 17603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:14.230026 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 17535
I20260807 18:33:14.230211 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:14.230758 27655 external_mini_cluster.cc:1428] Setting key 06a914e98f154ee58843bdbe688480fb
I20260807 18:33:14.241957 17603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395: Bootstrap starting.
I20260807 18:33:14.244082 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:14.232575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33959 (local address 127.0.0.1:33673)
0807 18:33:14.233400 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:14.233414 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:14.233455 (+ 41us) server_negotiation.cc:407] Connection header received
0807 18:33:14.235074 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:14.235098 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:14.235547 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:14.235839 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:14.236412 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:14.237676 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:14.239688 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:14.240369 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:14.240504 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:14.241763 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:14.241770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:14.241782 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:14.242355 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:14.242440 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:14.242469 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:14.242572 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:14.242796 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 18:33:14.243052 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":403,"threads_started":1}
I20260807 18:33:14.245301 17603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:14.246608 17603 log.cc:824] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:14.247936 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:14.249437 17603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395: No bootstrap required, opened a new log
I20260807 18:33:14.258476 17603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cabc4895284c38a95490e7a80a0395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33673 } }
I20260807 18:33:14.258901 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:14.252277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38483 (local address 127.0.0.1:33673)
0807 18:33:14.252461 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:14.252468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:14.252920 (+ 452us) server_negotiation.cc:407] Connection header received
0807 18:33:14.253065 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:14.253074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:14.253145 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:14.253247 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:14.253754 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:14.254909 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:14.256280 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:14.257050 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:14.257178 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:14.258154 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:14.258163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:14.258169 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:14.258188 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:14.258238 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:14.258261 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:14.258351 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:14.258494 (+ 143us) server_negotiation.cc:299] Negotiation successful
0807 18:33:14.258594 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 18:33:14.258948 17603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:14.259069 17603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cabc4895284c38a95490e7a80a0395, State: Initialized, Role: FOLLOWER
I20260807 18:33:14.259779 17603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cabc4895284c38a95490e7a80a0395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33673 } }
I20260807 18:33:14.260102 17603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:14.260283 17603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:14.260458 17603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:14.262630 17603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cabc4895284c38a95490e7a80a0395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33673 } }
I20260807 18:33:14.263167 17603 leader_election.cc:302] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70cabc4895284c38a95490e7a80a0395; no voters:
I20260807 18:33:14.263754 17603 leader_election.cc:288] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:14.263911 17614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:14.265084 17614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [term 1 LEADER]: Becoming Leader. State: Replica: 70cabc4895284c38a95490e7a80a0395, State: Running, Role: LEADER
I20260807 18:33:14.265657 17614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cabc4895284c38a95490e7a80a0395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33673 } }
I20260807 18:33:14.266511 17603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:14.268419 17615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70cabc4895284c38a95490e7a80a0395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cabc4895284c38a95490e7a80a0395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33673 } } }
I20260807 18:33:14.268796 17615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:14.269093 17617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70cabc4895284c38a95490e7a80a0395. Latest consensus state: current_term: 1 leader_uuid: "70cabc4895284c38a95490e7a80a0395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cabc4895284c38a95490e7a80a0395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33673 } } }
I20260807 18:33:14.269441 17617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:14.271003 17620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:14.275671 17620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:14.283747 17620 catalog_manager.cc:1347] Generated new cluster ID: 757084887ed74e34a13d5d04a248b27f
I20260807 18:33:14.283883 17620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:14.297755 17620 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:14.298868 17620 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:14.314303 17620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70cabc4895284c38a95490e7a80a0395: Generated new TSK 0
I20260807 18:33:14.315236 17620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:14.324465 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17535
W20260807 18:33:14.325784 17598 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33959
W20260807 18:33:14.326632 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17535: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:14.326797 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 17535
2026-08-07T18:33:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2074 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:14 dist-test-slave-1xrh krb5kdc[17637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:14 dist-test-slave-1xrh krb5kdc[17637](info): set up 2 sockets
Aug 07 18:33:14 dist-test-slave-1xrh krb5kdc[17637](info): commencing operation
krb5kdc: starting...
W20260807 18:33:15.497255 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:15 dist-test-slave-1xrh krb5kdc[17637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:15.806125 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:16.144824 17650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:16.145104 17650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:16.145287 17650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:16.152343 17650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:16.152469 17650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:16.152570 17650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:16.152670 17650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:16.152753 17650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:16.162385 17650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:16.163748 17650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:16.165104 17650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:16.176810 17656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:16.176860 17657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:16.176777 17659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:16.177604 17650 server_base.cc:1034] running on GCE node
I20260807 18:33:16.178412 17650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:16.179936 17650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:16.181198 17650 hybrid_clock.cc:648] HybridClock initialized: now 1786127596181162 us; error 49 us; skew 500 ppm
I20260807 18:33:16.181583 17650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:16 dist-test-slave-1xrh krb5kdc[17637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:16.185047 17650 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:16.186578 17650 webserver.cc:460] Webserver started at http://127.0.0.1:38963/ using document root <none> and password file <none>
I20260807 18:33:16.187345 17650 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:16.187494 17650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:16.187907 17650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:16.190946 17650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6"
format_stamp: "Formatted at 2026-08-07 18:33:16 on dist-test-slave-1xrh"
server_key: "94901ac543f29eef550aa9cd9ba9a025"
server_key_iv: "9094df53cfdec7a4443a12f21417a596"
server_key_version: "encryptionkey@0"
I20260807 18:33:16.191785 17650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6"
format_stamp: "Formatted at 2026-08-07 18:33:16 on dist-test-slave-1xrh"
server_key: "94901ac543f29eef550aa9cd9ba9a025"
server_key_iv: "9094df53cfdec7a4443a12f21417a596"
server_key_version: "encryptionkey@0"
I20260807 18:33:16.196964 17650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:33:16.200470 17666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:16.201792 17650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:33:16.202010 17650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6"
format_stamp: "Formatted at 2026-08-07 18:33:16 on dist-test-slave-1xrh"
server_key: "94901ac543f29eef550aa9cd9ba9a025"
server_key_iv: "9094df53cfdec7a4443a12f21417a596"
server_key_version: "encryptionkey@0"
I20260807 18:33:16.202217 17650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:16.208022 17650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:16.211854 17650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:16.212153 17650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:16.230840 17650 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44373
I20260807 18:33:16.230934 17721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44373 every 8 connection(s)
I20260807 18:33:16.232338 17650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:16.235770 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 17650
I20260807 18:33:16.235992 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:16.236181 17722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:16.236578 27655 external_mini_cluster.cc:1428] Setting key beba30ef69d8b4c57f2083e7b1838a0f
I20260807 18:33:16.245025 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:16.238484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46613 (local address 127.0.0.1:44373)
0807 18:33:16.239419 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:16.239435 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:16.239495 (+ 60us) server_negotiation.cc:407] Connection header received
0807 18:33:16.241196 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:16.241222 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:16.241649 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:16.242076 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:16.243083 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:16.243091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:16.243106 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:16.243151 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:16.243227 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:16.243257 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:16.243478 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:16.243615 (+ 137us) server_negotiation.cc:299] Negotiation successful
0807 18:33:16.243814 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":396,"threads_started":1}
I20260807 18:33:16.248847 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:16.250869 17722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6: Bootstrap starting.
I20260807 18:33:16.254953 17722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:16.256342 17722 log.cc:824] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:16.256897 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:16.254103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57247 (local address 127.0.0.1:44373)
0807 18:33:16.254338 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:16.254344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:16.254701 (+ 357us) server_negotiation.cc:407] Connection header received
0807 18:33:16.254875 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:16.254879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:16.254932 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:16.255022 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:16.256078 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:16.256083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:16.256086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:16.256101 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:16.256144 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:16.256159 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:16.256252 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:16.256431 (+ 179us) server_negotiation.cc:299] Negotiation successful
0807 18:33:16.256513 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:33:16.259312 17722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6: No bootstrap required, opened a new log
I20260807 18:33:16.266403 17722 raft_consensus.cc:348] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260807 18:33:16.266767 17722 raft_consensus.cc:374] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:16.266873 17722 raft_consensus.cc:729] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32da38ebd4c4dd19cb4b077f5c03fa6, State: Initialized, Role: FOLLOWER
I20260807 18:33:16.267467 17722 consensus_queue.cc:260] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260807 18:33:16.267714 17722 raft_consensus.cc:388] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:16.267853 17722 raft_consensus.cc:482] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:16.268035 17722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:16.270100 17722 raft_consensus.cc:504] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260807 18:33:16.270569 17722 leader_election.cc:302] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b32da38ebd4c4dd19cb4b077f5c03fa6; no voters:
I20260807 18:33:16.271198 17722 leader_election.cc:288] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:16.271505 17732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:16.272516 17732 raft_consensus.cc:686] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [term 1 LEADER]: Becoming Leader. State: Replica: b32da38ebd4c4dd19cb4b077f5c03fa6, State: Running, Role: LEADER
I20260807 18:33:16.273065 17732 consensus_queue.cc:237] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260807 18:33:16.273437 17722 sys_catalog.cc:564] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:16.275878 17733 sys_catalog.cc:455] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } } }
I20260807 18:33:16.276273 17733 sys_catalog.cc:458] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:16.276768 17734 sys_catalog.cc:455] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b32da38ebd4c4dd19cb4b077f5c03fa6. Latest consensus state: current_term: 1 leader_uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b32da38ebd4c4dd19cb4b077f5c03fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } } }
I20260807 18:33:16.277077 17734 sys_catalog.cc:458] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:16.278379 17739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:16.283833 17739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:16.292230 17739 catalog_manager.cc:1347] Generated new cluster ID: d017b64ceb2243ad8c1a2c33780d40e1
I20260807 18:33:16.292335 17739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:16.321894 17739 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:16.322379 17739 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:16.328718 17739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b32da38ebd4c4dd19cb4b077f5c03fa6: Generated new TSK 0
I20260807 18:33:16.329455 17739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:16.347040 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17650
W20260807 18:33:16.348222 17717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46613
W20260807 18:33:16.348980 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17650: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:16.349130 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 17650
2026-08-07T18:33:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:16 dist-test-slave-1xrh krb5kdc[17754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:16 dist-test-slave-1xrh krb5kdc[17754](info): set up 2 sockets
Aug 07 18:33:16 dist-test-slave-1xrh krb5kdc[17754](info): commencing operation
krb5kdc: starting...
W20260807 18:33:17.557758 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:17 dist-test-slave-1xrh krb5kdc[17754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:17.870694 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:18.213106 17769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:18.213387 17769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:18.213482 17769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:18.220556 17769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:18.220660 17769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:18.220706 17769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:18.220775 17769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:18.220813 17769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:18.230340 17769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:18.231623 17769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:18.232908 17769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:18.244679 17774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:18.244702 17777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:18.244752 17775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:18.245435 17769 server_base.cc:1034] running on GCE node
I20260807 18:33:18.246141 17769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:18.247617 17769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:18.248839 17769 hybrid_clock.cc:648] HybridClock initialized: now 1786127598248784 us; error 43 us; skew 500 ppm
I20260807 18:33:18.249207 17769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:18 dist-test-slave-1xrh krb5kdc[17754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:18.252504 17769 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:18.253988 17769 webserver.cc:460] Webserver started at http://127.0.0.1:41611/ using document root <none> and password file <none>
I20260807 18:33:18.254679 17769 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:18.254830 17769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:18.255199 17769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:18.258174 17769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "3b6fa74a629d4f6384f4e56faebeef84"
format_stamp: "Formatted at 2026-08-07 18:33:18 on dist-test-slave-1xrh"
server_key: "07d7950ab7440dcafde10ff54e409efb"
server_key_iv: "7f75224b87d90d39f1ec58bea586d936"
server_key_version: "encryptionkey@0"
I20260807 18:33:18.259045 17769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "3b6fa74a629d4f6384f4e56faebeef84"
format_stamp: "Formatted at 2026-08-07 18:33:18 on dist-test-slave-1xrh"
server_key: "07d7950ab7440dcafde10ff54e409efb"
server_key_iv: "7f75224b87d90d39f1ec58bea586d936"
server_key_version: "encryptionkey@0"
I20260807 18:33:18.264112 17769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:33:18.267516 17784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:18.268769 17769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:33:18.268975 17769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "3b6fa74a629d4f6384f4e56faebeef84"
format_stamp: "Formatted at 2026-08-07 18:33:18 on dist-test-slave-1xrh"
server_key: "07d7950ab7440dcafde10ff54e409efb"
server_key_iv: "7f75224b87d90d39f1ec58bea586d936"
server_key_version: "encryptionkey@0"
I20260807 18:33:18.269200 17769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:18.293680 17769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:18.298005 17769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:18.298331 17769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:18.316668 17769 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33761
I20260807 18:33:18.316712 17839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33761 every 8 connection(s)
I20260807 18:33:18.318023 17769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:18.320137 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 17769
I20260807 18:33:18.320338 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:18.320858 27655 external_mini_cluster.cc:1428] Setting key 2dfdbf209d6e27e0d7cb25df646ab4d1
I20260807 18:33:18.322131 17840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:18.337199 17840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84: Bootstrap starting.
Aug 07 18:33:18 dist-test-slave-1xrh krb5kdc[17754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 18:33:18.340998 17840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:18.342139 17840 log.cc:824] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:18.345598 17840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84: No bootstrap required, opened a new log
I20260807 18:33:18.346500 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:18.322909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:43921 (local address 127.0.0.1:33761)
0807 18:33:18.323779 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:18.323808 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:18.323861 (+ 53us) server_negotiation.cc:407] Connection header received
0807 18:33:18.325649 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:18.325672 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:18.326383 (+ 711us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:18.326907 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:18.328893 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:18.330411 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:18.332234 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:18.333103 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:18.333263 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:18.338641 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:18.338650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:18.338662 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:18.338705 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:18.341959 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:18.343022 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:18.343032 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:18.343043 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:18.343290 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:18.343822 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:18.343825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:18.343827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:18.344354 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:18.344591 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:18.345149 (+ 558us) server_negotiation.cc:299] Negotiation successful
0807 18:33:18.345447 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":400,"threads_started":1}
I20260807 18:33:18.350575 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:18.356177 17840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6fa74a629d4f6384f4e56faebeef84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260807 18:33:18.356673 17840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:18.356805 17840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6fa74a629d4f6384f4e56faebeef84, State: Initialized, Role: FOLLOWER
I20260807 18:33:18.357527 17840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6fa74a629d4f6384f4e56faebeef84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260807 18:33:18.357841 17840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:18.357990 17840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:18.358199 17840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:18.361296 17840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6fa74a629d4f6384f4e56faebeef84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260807 18:33:18.361927 17840 leader_election.cc:302] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b6fa74a629d4f6384f4e56faebeef84; no voters:
I20260807 18:33:18.362643 17840 leader_election.cc:288] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:18.362751 17850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:18.365092 17840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:18.368675 17850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [term 1 LEADER]: Becoming Leader. State: Replica: 3b6fa74a629d4f6384f4e56faebeef84, State: Running, Role: LEADER
I20260807 18:33:18.369341 17850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6fa74a629d4f6384f4e56faebeef84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260807 18:33:18.374944 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:18.355965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:49459 (local address 127.0.0.1:33761)
0807 18:33:18.356195 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:18.356204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:18.356620 (+ 416us) server_negotiation.cc:407] Connection header received
0807 18:33:18.356836 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:18.356846 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:18.356976 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:18.357056 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:18.359050 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:18.359990 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:18.361776 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:18.362304 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:18.362468 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:18.365847 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:18.365864 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:18.365872 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:18.365928 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:18.370281 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:18.371516 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:18.371525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:18.371531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:18.371740 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:18.372564 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:18.372573 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:18.372579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:18.373077 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:18.373486 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:18.374120 (+ 634us) server_negotiation.cc:299] Negotiation successful
0807 18:33:18.374380 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 18:33:18.379504 17852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b6fa74a629d4f6384f4e56faebeef84. Latest consensus state: current_term: 1 leader_uuid: "3b6fa74a629d4f6384f4e56faebeef84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6fa74a629d4f6384f4e56faebeef84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } } }
I20260807 18:33:18.379894 17852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:18.381937 17851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b6fa74a629d4f6384f4e56faebeef84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6fa74a629d4f6384f4e56faebeef84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } } }
I20260807 18:33:18.382385 17851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:18.383215 17860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:18.387686 17860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:18.396245 17860 catalog_manager.cc:1347] Generated new cluster ID: 2bddc0713583409a9972af710979df58
I20260807 18:33:18.396347 17860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:18.411769 17860 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:18.413558 17860 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:18.429405 17860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b6fa74a629d4f6384f4e56faebeef84: Generated new TSK 0
I20260807 18:33:18.430249 17860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:18.442035 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17769
I20260807 18:33:18.443271 17834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:18.536216 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 17769
W20260807 18:33:18.558918 17752 connection.cc:535] client connection to 127.0.0.1:33761 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33761): Connection reset by peer (error 104)
2026-08-07T18:33:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2188 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:18 dist-test-slave-1xrh krb5kdc[17874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:18 dist-test-slave-1xrh krb5kdc[17874](info): set up 2 sockets
Aug 07 18:33:18 dist-test-slave-1xrh krb5kdc[17874](info): commencing operation
krb5kdc: starting...
W20260807 18:33:19.709972 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:19 dist-test-slave-1xrh krb5kdc[17874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:20.032748 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:20.376578 17889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:20.376885 17889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:20.376986 17889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:20.384241 17889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:20.384351 17889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:20.384397 17889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:20.384459 17889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:20.384503 17889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:20.394246 17889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:20.395498 17889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:20.396751 17889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:20.411659 17894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:20.411607 17895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:20.411685 17897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:20.412457 17889 server_base.cc:1034] running on GCE node
I20260807 18:33:20.413246 17889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:20.414676 17889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:20.415943 17889 hybrid_clock.cc:648] HybridClock initialized: now 1786127600415902 us; error 40 us; skew 500 ppm
I20260807 18:33:20.416298 17889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:20 dist-test-slave-1xrh krb5kdc[17874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:20.427160 17889 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:20.428752 17889 webserver.cc:460] Webserver started at http://127.0.0.1:32999/ using document root <none> and password file <none>
I20260807 18:33:20.429456 17889 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:20.429606 17889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:20.429924 17889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:20.433032 17889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "b077ebbbaa10403c94c15d6bb6d807d3"
format_stamp: "Formatted at 2026-08-07 18:33:20 on dist-test-slave-1xrh"
server_key: "c4ba6a84e9cb0bd3484c23d89478464f"
server_key_iv: "6677fafa6987efa71d1b0a4bb6b358d7"
server_key_version: "encryptionkey@0"
I20260807 18:33:20.433826 17889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "b077ebbbaa10403c94c15d6bb6d807d3"
format_stamp: "Formatted at 2026-08-07 18:33:20 on dist-test-slave-1xrh"
server_key: "c4ba6a84e9cb0bd3484c23d89478464f"
server_key_iv: "6677fafa6987efa71d1b0a4bb6b358d7"
server_key_version: "encryptionkey@0"
I20260807 18:33:20.438983 17889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 18:33:20.442368 17904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:20.443646 17889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:33:20.443825 17889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "b077ebbbaa10403c94c15d6bb6d807d3"
format_stamp: "Formatted at 2026-08-07 18:33:20 on dist-test-slave-1xrh"
server_key: "c4ba6a84e9cb0bd3484c23d89478464f"
server_key_iv: "6677fafa6987efa71d1b0a4bb6b358d7"
server_key_version: "encryptionkey@0"
I20260807 18:33:20.444031 17889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:20.456851 17889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:20.460733 17889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:20.461014 17889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:20.480050 17889 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41887
I20260807 18:33:20.480125 17958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41887 every 8 connection(s)
I20260807 18:33:20.481374 17889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:20.483969 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 17889
I20260807 18:33:20.484158 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:20.484697 27655 external_mini_cluster.cc:1428] Setting key ee9040aec3e121f9626609f2be526c65
I20260807 18:33:20.485445 17959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:20.497203 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:20.486537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:59045 (local address 127.0.0.1:41887)
0807 18:33:20.487316 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:20.487327 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:20.487371 (+ 44us) server_negotiation.cc:407] Connection header received
0807 18:33:20.488884 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:20.488902 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:20.489293 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:20.489683 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:20.490228 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:20.491590 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:20.492857 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:20.493500 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:20.493676 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:20.494997 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:20.495008 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:20.495020 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:20.495610 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:20.495683 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:20.495711 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:20.495887 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:20.496179 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 18:33:20.496327 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":324,"threads_started":1}
I20260807 18:33:20.497905 17959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3: Bootstrap starting.
I20260807 18:33:20.501039 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:20.501363 17959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:20.502331 17959 log.cc:824] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:20.504978 17959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3: No bootstrap required, opened a new log
I20260807 18:33:20.512200 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:20.505330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:58789 (local address 127.0.0.1:41887)
0807 18:33:20.505516 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:20.505522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:20.506068 (+ 546us) server_negotiation.cc:407] Connection header received
0807 18:33:20.506201 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:20.506207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:20.506280 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:20.506401 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:20.506908 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:20.508129 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:20.509481 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:20.510243 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:20.510386 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:20.511442 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:20.511449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:20.511455 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:20.511475 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:20.511538 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:20.511558 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:20.511684 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:20.511833 (+ 149us) server_negotiation.cc:299] Negotiation successful
0807 18:33:20.511889 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 18:33:20.514276 17959 raft_consensus.cc:348] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077ebbbaa10403c94c15d6bb6d807d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41887 } }
I20260807 18:33:20.514599 17959 raft_consensus.cc:374] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:20.514671 17959 raft_consensus.cc:729] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b077ebbbaa10403c94c15d6bb6d807d3, State: Initialized, Role: FOLLOWER
I20260807 18:33:20.515210 17959 consensus_queue.cc:260] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077ebbbaa10403c94c15d6bb6d807d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41887 } }
I20260807 18:33:20.515421 17959 raft_consensus.cc:388] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:20.515523 17959 raft_consensus.cc:482] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:20.515650 17959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:20.517683 17959 raft_consensus.cc:504] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077ebbbaa10403c94c15d6bb6d807d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41887 } }
I20260807 18:33:20.518108 17959 leader_election.cc:302] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b077ebbbaa10403c94c15d6bb6d807d3; no voters:
I20260807 18:33:20.518716 17959 leader_election.cc:288] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:20.519016 17970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:20.520244 17970 raft_consensus.cc:686] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [term 1 LEADER]: Becoming Leader. State: Replica: b077ebbbaa10403c94c15d6bb6d807d3, State: Running, Role: LEADER
I20260807 18:33:20.520742 17970 consensus_queue.cc:237] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077ebbbaa10403c94c15d6bb6d807d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41887 } }
I20260807 18:33:20.521086 17959 sys_catalog.cc:564] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:20.523625 17971 sys_catalog.cc:455] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b077ebbbaa10403c94c15d6bb6d807d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077ebbbaa10403c94c15d6bb6d807d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41887 } } }
I20260807 18:33:20.523577 17972 sys_catalog.cc:455] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b077ebbbaa10403c94c15d6bb6d807d3. Latest consensus state: current_term: 1 leader_uuid: "b077ebbbaa10403c94c15d6bb6d807d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077ebbbaa10403c94c15d6bb6d807d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41887 } } }
I20260807 18:33:20.524096 17972 sys_catalog.cc:458] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:20.524096 17971 sys_catalog.cc:458] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:20.525750 17977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:20.533840 17977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:20.542935 17977 catalog_manager.cc:1347] Generated new cluster ID: 5f92be3686e4477e845c022f9ef5b4fd
I20260807 18:33:20.543155 17977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:20.554050 17977 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:20.555389 17977 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:20.567660 17977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b077ebbbaa10403c94c15d6bb6d807d3: Generated new TSK 0
I20260807 18:33:20.568554 17977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:20.574879 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17889
W20260807 18:33:20.576556 17954 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.186:59045
W20260807 18:33:20.577396 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 17889: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:20.577554 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 17889
W20260807 18:33:20.600054 17872 connection.cc:535] client connection to 127.0.0.1:41887 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41887): Connection reset by peer (error 104)
2026-08-07T18:33:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2041 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:20 dist-test-slave-1xrh krb5kdc[17994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:20 dist-test-slave-1xrh krb5kdc[17994](info): set up 2 sockets
Aug 07 18:33:20 dist-test-slave-1xrh krb5kdc[17994](info): commencing operation
krb5kdc: starting...
W20260807 18:33:21.853724 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:22 dist-test-slave-1xrh krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:22.179874 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:22.521560 18007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:22.521893 18007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:22.522063 18007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:22.529167 18007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:22.529320 18007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:22.529428 18007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:22.529524 18007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:22.529604 18007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:22.539837 18007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:22.541129 18007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:22.542395 18007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:22.554291 18015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:22.554347 18012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:22.554462 18013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:22.555267 18007 server_base.cc:1034] running on GCE node
I20260807 18:33:22.556025 18007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:22.557335 18007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:22.558568 18007 hybrid_clock.cc:648] HybridClock initialized: now 1786127602558521 us; error 49 us; skew 500 ppm
I20260807 18:33:22.558907 18007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:22 dist-test-slave-1xrh krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:22.562081 18007 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:22.563578 18007 webserver.cc:460] Webserver started at http://127.0.0.1:39949/ using document root <none> and password file <none>
I20260807 18:33:22.564265 18007 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:22.564405 18007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:22.564700 18007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:22.568065 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "d2e334b76fc24cda87ca28041b48f023"
format_stamp: "Formatted at 2026-08-07 18:33:22 on dist-test-slave-1xrh"
server_key: "c8f067676de7ee967d8874006489a4b0"
server_key_iv: "68a965fbe60d7f0ace308ddafbfa0b12"
server_key_version: "encryptionkey@0"
I20260807 18:33:22.568856 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "d2e334b76fc24cda87ca28041b48f023"
format_stamp: "Formatted at 2026-08-07 18:33:22 on dist-test-slave-1xrh"
server_key: "c8f067676de7ee967d8874006489a4b0"
server_key_iv: "68a965fbe60d7f0ace308ddafbfa0b12"
server_key_version: "encryptionkey@0"
I20260807 18:33:22.575206 18007 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 18:33:22.578495 18022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:22.579825 18007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 18:33:22.580029 18007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "d2e334b76fc24cda87ca28041b48f023"
format_stamp: "Formatted at 2026-08-07 18:33:22 on dist-test-slave-1xrh"
server_key: "c8f067676de7ee967d8874006489a4b0"
server_key_iv: "68a965fbe60d7f0ace308ddafbfa0b12"
server_key_version: "encryptionkey@0"
I20260807 18:33:22.580240 18007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:22.585688 18007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:22.589473 18007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:22.589712 18007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:22.608034 18007 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36127
I20260807 18:33:22.608101 18075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36127 every 8 connection(s)
I20260807 18:33:22.609393 18007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:22.610986 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 18007
I20260807 18:33:22.611240 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:22.611829 27655 external_mini_cluster.cc:1428] Setting key e2da4d4d47cdc4bc57a25e2a4ea38e9a
I20260807 18:33:22.614030 18076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:22.619514 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:22.613667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:46037 (local address 127.0.0.1:36127)
0807 18:33:22.614469 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:22.614479 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:22.614520 (+ 41us) server_negotiation.cc:407] Connection header received
0807 18:33:22.616184 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:22.616204 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:22.616525 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:22.616843 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:22.617950 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:22.617956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:22.617967 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:22.617981 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:22.618056 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:22.618075 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:22.618242 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:22.618339 (+ 97us) server_negotiation.cc:299] Negotiation successful
0807 18:33:22.618551 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":373,"threads_started":1}
I20260807 18:33:22.623070 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:22.625810 18076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023: Bootstrap starting.
I20260807 18:33:22.629391 18076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:22.629998 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:22.627490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:55389 (local address 127.0.0.1:36127)
0807 18:33:22.627699 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:22.627705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:22.628009 (+ 304us) server_negotiation.cc:407] Connection header received
0807 18:33:22.628190 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:22.628193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:22.628236 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:22.628313 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:22.629261 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:22.629265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:22.629268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:22.629282 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:22.629320 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:22.629334 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:22.629437 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:22.629632 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 18:33:22.629682 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 18:33:22.630688 18076 log.cc:824] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:22.633386 18076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023: No bootstrap required, opened a new log
I20260807 18:33:22.641589 18076 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e334b76fc24cda87ca28041b48f023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } }
I20260807 18:33:22.642028 18076 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:22.642112 18076 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e334b76fc24cda87ca28041b48f023, State: Initialized, Role: FOLLOWER
I20260807 18:33:22.642692 18076 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e334b76fc24cda87ca28041b48f023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } }
I20260807 18:33:22.642942 18076 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:22.643057 18076 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:22.643230 18076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:22.645896 18076 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e334b76fc24cda87ca28041b48f023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } }
I20260807 18:33:22.646409 18076 leader_election.cc:302] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e334b76fc24cda87ca28041b48f023; no voters:
I20260807 18:33:22.646945 18076 leader_election.cc:288] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:22.647104 18087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:22.647820 18087 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [term 1 LEADER]: Becoming Leader. State: Replica: d2e334b76fc24cda87ca28041b48f023, State: Running, Role: LEADER
I20260807 18:33:22.648327 18087 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e334b76fc24cda87ca28041b48f023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } }
I20260807 18:33:22.649322 18076 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:22.652447 18088 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2e334b76fc24cda87ca28041b48f023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e334b76fc24cda87ca28041b48f023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } } }
I20260807 18:33:22.653403 18088 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:22.652699 18090 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e334b76fc24cda87ca28041b48f023. Latest consensus state: current_term: 1 leader_uuid: "d2e334b76fc24cda87ca28041b48f023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e334b76fc24cda87ca28041b48f023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } } }
I20260807 18:33:22.654379 18090 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:22.655319 18096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:22.660903 18096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:22.670329 18096 catalog_manager.cc:1347] Generated new cluster ID: b7632626c59d4564ba123c261d9dfe88
I20260807 18:33:22.670485 18096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:22.693481 18096 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:22.693990 18096 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:22.704005 18096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2e334b76fc24cda87ca28041b48f023: Generated new TSK 0
I20260807 18:33:22.705050 18096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:22.711181 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18007
W20260807 18:33:22.712631 18070 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.186:46037
W20260807 18:33:22.713531 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18007: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:22.713680 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 18007
2026-08-07T18:33:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2143 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:22 dist-test-slave-1xrh krb5kdc[18109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:22 dist-test-slave-1xrh krb5kdc[18109](info): set up 2 sockets
Aug 07 18:33:22 dist-test-slave-1xrh krb5kdc[18109](info): commencing operation
krb5kdc: starting...
W20260807 18:33:23.917456 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:24 dist-test-slave-1xrh krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:24.231643 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:24.572481 18124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:24.572764 18124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:24.572865 18124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:24.579773 18124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:24.579893 18124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:24.579949 18124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:24.580013 18124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:24.580060 18124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:24.589486 18124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:24.590674 18124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:24.591933 18124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:24.603667 18129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:24.603654 18130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:24.603883 18133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:24.604506 18124 server_base.cc:1034] running on GCE node
I20260807 18:33:24.605227 18124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:24.606592 18124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:24.607810 18124 hybrid_clock.cc:648] HybridClock initialized: now 1786127604607742 us; error 64 us; skew 500 ppm
I20260807 18:33:24.608178 18124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:24 dist-test-slave-1xrh krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:24.611392 18124 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:24.612802 18124 webserver.cc:460] Webserver started at http://127.0.0.1:33629/ using document root <none> and password file <none>
I20260807 18:33:24.613423 18124 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:24.613524 18124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:24.613772 18124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:24.616747 18124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "f6672db995054299a80c42e04f52ce49"
format_stamp: "Formatted at 2026-08-07 18:33:24 on dist-test-slave-1xrh"
server_key: "93adeee6bb1fa6f725ad1c92ce976515"
server_key_iv: "40040639092093367d25fb3be9490ed2"
server_key_version: "encryptionkey@0"
I20260807 18:33:24.617472 18124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "f6672db995054299a80c42e04f52ce49"
format_stamp: "Formatted at 2026-08-07 18:33:24 on dist-test-slave-1xrh"
server_key: "93adeee6bb1fa6f725ad1c92ce976515"
server_key_iv: "40040639092093367d25fb3be9490ed2"
server_key_version: "encryptionkey@0"
I20260807 18:33:24.622467 18124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 18:33:24.625833 18140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:24.627276 18124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:33:24.627471 18124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "f6672db995054299a80c42e04f52ce49"
format_stamp: "Formatted at 2026-08-07 18:33:24 on dist-test-slave-1xrh"
server_key: "93adeee6bb1fa6f725ad1c92ce976515"
server_key_iv: "40040639092093367d25fb3be9490ed2"
server_key_version: "encryptionkey@0"
I20260807 18:33:24.627638 18124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:24.650543 18124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:24.654865 18124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:24.655186 18124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:24.674089 18124 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38099
I20260807 18:33:24.674124 18194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38099 every 8 connection(s)
I20260807 18:33:24.675506 18124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:24.679450 18195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:24.680317 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 18124
I20260807 18:33:24.680506 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:24.681092 27655 external_mini_cluster.cc:1428] Setting key b987c4cc91358cdd0f8736b8e4bd4f3f
I20260807 18:33:24.693531 18195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49: Bootstrap starting.
Aug 07 18:33:24 dist-test-slave-1xrh krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 18:33:24.697014 18195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:24.698199 18195 log.cc:824] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:24.701167 18195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49: No bootstrap required, opened a new log
I20260807 18:33:24.705292 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:24.682874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:56071 (local address 127.0.0.1:38099)
0807 18:33:24.683648 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:24.683659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:24.683699 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:33:24.685747 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:24.685770 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:24.686235 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:24.686566 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:24.688351 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:24.689643 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:24.691510 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:24.692156 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:24.692283 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:24.697208 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:24.697217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:24.697231 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:24.697277 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:24.700641 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:24.701459 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:24.701469 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:24.701479 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:24.701711 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:24.702137 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:24.702142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:24.702146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:24.702819 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:24.703205 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:24.703577 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 18:33:24.703951 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":383,"threads_started":1}
I20260807 18:33:24.709142 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:24.710950 18195 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6672db995054299a80c42e04f52ce49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260807 18:33:24.711390 18195 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:24.711504 18195 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6672db995054299a80c42e04f52ce49, State: Initialized, Role: FOLLOWER
I20260807 18:33:24.712143 18195 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6672db995054299a80c42e04f52ce49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260807 18:33:24.712479 18195 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:24.712641 18195 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:24.712846 18195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:24.715865 18195 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6672db995054299a80c42e04f52ce49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260807 18:33:24.716334 18195 leader_election.cc:302] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6672db995054299a80c42e04f52ce49; no voters:
I20260807 18:33:24.716905 18195 leader_election.cc:288] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:24.717056 18205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:24.718811 18205 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [term 1 LEADER]: Becoming Leader. State: Replica: f6672db995054299a80c42e04f52ce49, State: Running, Role: LEADER
I20260807 18:33:24.719241 18195 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:24.719507 18205 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6672db995054299a80c42e04f52ce49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260807 18:33:24.722182 18208 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6672db995054299a80c42e04f52ce49. Latest consensus state: current_term: 1 leader_uuid: "f6672db995054299a80c42e04f52ce49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6672db995054299a80c42e04f52ce49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } } }
I20260807 18:33:24.722657 18208 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:24.730587 18211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:24.730684 18207 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6672db995054299a80c42e04f52ce49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6672db995054299a80c42e04f52ce49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } } }
I20260807 18:33:24.731014 18207 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:24.736193 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:24.714618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:46661 (local address 127.0.0.1:38099)
0807 18:33:24.714807 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:24.714814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:24.715247 (+ 433us) server_negotiation.cc:407] Connection header received
0807 18:33:24.715388 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:24.715393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:24.715468 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:24.715572 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:24.717192 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:24.718452 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:24.720336 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:24.721250 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:24.721411 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:24.725872 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:24.725882 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:24.725889 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:24.725942 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:24.730777 (+ 4835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:24.732055 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:24.732063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:24.732069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:24.732259 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:24.732936 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:24.732942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:24.732946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:24.733447 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:24.733757 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:24.735395 (+ 1638us) server_negotiation.cc:299] Negotiation successful
0807 18:33:24.735647 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 18:33:24.736465 18211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:24.746203 18211 catalog_manager.cc:1347] Generated new cluster ID: 874074f100a046619e6498086608dd4c
I20260807 18:33:24.746397 18211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:24.767395 18211 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:24.769281 18211 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:24.779971 18211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6672db995054299a80c42e04f52ce49: Generated new TSK 0
I20260807 18:33:24.780997 18211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:24.789459 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18124
I20260807 18:33:24.790951 18188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:24.886217 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 18124
W20260807 18:33:24.907797 18107 connection.cc:535] client connection to 127.0.0.1:38099 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38099): Connection reset by peer (error 104)
2026-08-07T18:33:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2168 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:25 dist-test-slave-1xrh krb5kdc[18230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:25 dist-test-slave-1xrh krb5kdc[18230](info): set up 2 sockets
Aug 07 18:33:25 dist-test-slave-1xrh krb5kdc[18230](info): commencing operation
krb5kdc: starting...
W20260807 18:33:26.085251 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.004s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:26 dist-test-slave-1xrh krb5kdc[18230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:26.406134 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:26.747552 18244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:26.747895 18244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:26.748051 18244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:26.755112 18244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:26.755285 18244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:26.755385 18244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:26.755492 18244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:26.755576 18244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:26.765343 18244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:26.766664 18244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:26.768004 18244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:26.780607 18249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:26.780604 18250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:26.780846 18252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:26.781502 18244 server_base.cc:1034] running on GCE node
I20260807 18:33:26.782277 18244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:26.783773 18244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:26.785012 18244 hybrid_clock.cc:648] HybridClock initialized: now 1786127606784973 us; error 28 us; skew 500 ppm
I20260807 18:33:26.785353 18244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:26 dist-test-slave-1xrh krb5kdc[18230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:26.794468 18244 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:26.796164 18244 webserver.cc:460] Webserver started at http://127.0.0.1:38741/ using document root <none> and password file <none>
I20260807 18:33:26.796886 18244 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:26.797030 18244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:26.797351 18244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:26.800446 18244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "3e19163b8e5f41a795ec609f2f342f04"
format_stamp: "Formatted at 2026-08-07 18:33:26 on dist-test-slave-1xrh"
server_key: "cb06034270ffb6fa471447380bf9567c"
server_key_iv: "81b3ba06cc58e7dab1bee5cb4b93b306"
server_key_version: "encryptionkey@0"
I20260807 18:33:26.801210 18244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "3e19163b8e5f41a795ec609f2f342f04"
format_stamp: "Formatted at 2026-08-07 18:33:26 on dist-test-slave-1xrh"
server_key: "cb06034270ffb6fa471447380bf9567c"
server_key_iv: "81b3ba06cc58e7dab1bee5cb4b93b306"
server_key_version: "encryptionkey@0"
I20260807 18:33:26.807999 18244 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 18:33:26.812942 18259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:26.814445 18244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260807 18:33:26.814615 18244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "3e19163b8e5f41a795ec609f2f342f04"
format_stamp: "Formatted at 2026-08-07 18:33:26 on dist-test-slave-1xrh"
server_key: "cb06034270ffb6fa471447380bf9567c"
server_key_iv: "81b3ba06cc58e7dab1bee5cb4b93b306"
server_key_version: "encryptionkey@0"
I20260807 18:33:26.814788 18244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:26.827486 18244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:26.831962 18244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:26.832199 18244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:26.850911 18244 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36781
I20260807 18:33:26.850934 18314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36781 every 8 connection(s)
I20260807 18:33:26.852424 18244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:26.856161 18315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:26.856547 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 18244
I20260807 18:33:26.856739 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:26.857239 27655 external_mini_cluster.cc:1428] Setting key e12c29685ad59cd06d3e6d1221d37c56
I20260807 18:33:26.870919 18315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04: Bootstrap starting.
I20260807 18:33:26.873333 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:26.859225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:59011 (local address 127.0.0.1:36781)
0807 18:33:26.860062 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:26.860077 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:26.860130 (+ 53us) server_negotiation.cc:407] Connection header received
0807 18:33:26.862920 (+ 2790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:26.862950 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:26.863493 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:26.863955 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:26.864447 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:26.866336 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:26.867911 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:26.869006 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:26.869174 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:26.870508 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:26.870518 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:26.870535 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:26.871254 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:26.871361 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:26.871401 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:26.871566 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:26.871848 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 18:33:26.871998 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":347,"threads_started":1}
I20260807 18:33:26.874902 18315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:26.876149 18315 log.cc:824] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:26.877179 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:26.878862 18315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04: No bootstrap required, opened a new log
I20260807 18:33:26.888360 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:26.881620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:48737 (local address 127.0.0.1:36781)
0807 18:33:26.881834 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:26.881840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:26.882158 (+ 318us) server_negotiation.cc:407] Connection header received
0807 18:33:26.882335 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:26.882338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:26.882405 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:26.882489 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:26.883087 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:26.883946 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:26.885786 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:26.886237 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:26.886327 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:26.887619 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:26.887624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:26.887627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:26.887643 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:26.887677 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:26.887691 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:26.887771 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:26.887994 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 18:33:26.888057 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 18:33:26.888429 18315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e19163b8e5f41a795ec609f2f342f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } }
I20260807 18:33:26.888873 18315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:26.889003 18315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e19163b8e5f41a795ec609f2f342f04, State: Initialized, Role: FOLLOWER
I20260807 18:33:26.889664 18315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e19163b8e5f41a795ec609f2f342f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } }
I20260807 18:33:26.889959 18315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:26.890102 18315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:26.890223 18315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:26.892392 18315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e19163b8e5f41a795ec609f2f342f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } }
I20260807 18:33:26.892803 18315 leader_election.cc:302] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e19163b8e5f41a795ec609f2f342f04; no voters:
I20260807 18:33:26.893306 18315 leader_election.cc:288] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:26.893446 18326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:26.894124 18326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [term 1 LEADER]: Becoming Leader. State: Replica: 3e19163b8e5f41a795ec609f2f342f04, State: Running, Role: LEADER
I20260807 18:33:26.894645 18326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e19163b8e5f41a795ec609f2f342f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } }
I20260807 18:33:26.895417 18315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:26.897459 18327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e19163b8e5f41a795ec609f2f342f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e19163b8e5f41a795ec609f2f342f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } } }
I20260807 18:33:26.897804 18327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:26.903031 18328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e19163b8e5f41a795ec609f2f342f04. Latest consensus state: current_term: 1 leader_uuid: "3e19163b8e5f41a795ec609f2f342f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e19163b8e5f41a795ec609f2f342f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } } }
I20260807 18:33:26.904327 18328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:26.904635 18332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:26.909610 18332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:26.918977 18332 catalog_manager.cc:1347] Generated new cluster ID: 40bd04e822fd4082ae9095ff58cbc08c
I20260807 18:33:26.919142 18332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:26.939345 18332 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:26.940886 18332 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:26.951359 18332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e19163b8e5f41a795ec609f2f342f04: Generated new TSK 0
I20260807 18:33:26.952324 18332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:26.965241 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18244
W20260807 18:33:26.967334 18310 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.186:59011
W20260807 18:33:26.968308 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18244: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:26.968483 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 18244
W20260807 18:33:26.989837 18228 connection.cc:535] client connection to 127.0.0.1:36781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36781): Connection reset by peer (error 104)
2026-08-07T18:33:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2078 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:27 dist-test-slave-1xrh krb5kdc[18349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:27 dist-test-slave-1xrh krb5kdc[18349](info): set up 2 sockets
Aug 07 18:33:27 dist-test-slave-1xrh krb5kdc[18349](info): commencing operation
krb5kdc: starting...
W20260807 18:33:28.129916 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:28 dist-test-slave-1xrh krb5kdc[18349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 18:33:28.440261 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:28.783730 18362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:28.784127 18362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:28.784245 18362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:28.792261 18362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:28.792380 18362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:28.792441 18362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:28.792510 18362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:28.792559 18362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:28.802930 18362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:28.804263 18362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:28.805552 18362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:28.817823 18368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:28.817792 18369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:28.818226 18371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:28.818686 18362 server_base.cc:1034] running on GCE node
I20260807 18:33:28.819499 18362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:28.820828 18362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:28.822050 18362 hybrid_clock.cc:648] HybridClock initialized: now 1786127608822007 us; error 27 us; skew 500 ppm
I20260807 18:33:28.822402 18362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:28 dist-test-slave-1xrh krb5kdc[18349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:28.825554 18362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 18:33:28.826900 18362 webserver.cc:460] Webserver started at http://127.0.0.1:44381/ using document root <none> and password file <none>
I20260807 18:33:28.827587 18362 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:28.827693 18362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:28.827947 18362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:28.830842 18362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "4d943141b9274926b089a8d47d9d07a0"
format_stamp: "Formatted at 2026-08-07 18:33:28 on dist-test-slave-1xrh"
server_key: "d1bba07f1e1a4b1cb9491a3d4f762be8"
server_key_iv: "e1fa7264d88c31b8eb1438fb57eb97fd"
server_key_version: "encryptionkey@0"
I20260807 18:33:28.831668 18362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "4d943141b9274926b089a8d47d9d07a0"
format_stamp: "Formatted at 2026-08-07 18:33:28 on dist-test-slave-1xrh"
server_key: "d1bba07f1e1a4b1cb9491a3d4f762be8"
server_key_iv: "e1fa7264d88c31b8eb1438fb57eb97fd"
server_key_version: "encryptionkey@0"
I20260807 18:33:28.836649 18362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:33:28.839887 18378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:28.841197 18362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260807 18:33:28.841364 18362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "4d943141b9274926b089a8d47d9d07a0"
format_stamp: "Formatted at 2026-08-07 18:33:28 on dist-test-slave-1xrh"
server_key: "d1bba07f1e1a4b1cb9491a3d4f762be8"
server_key_iv: "e1fa7264d88c31b8eb1438fb57eb97fd"
server_key_version: "encryptionkey@0"
I20260807 18:33:28.841531 18362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:28.847057 18362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:28.850903 18362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:28.851187 18362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:28.869781 18362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42809
I20260807 18:33:28.869931 18433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42809 every 8 connection(s)
I20260807 18:33:28.871238 18362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:28.875463 18434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:28.879554 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 18362
I20260807 18:33:28.879777 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:28.880395 27655 external_mini_cluster.cc:1428] Setting key fb918a553430613693633017655c01c2
I20260807 18:33:28.888304 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:28.882300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:36597 (local address 127.0.0.1:42809)
0807 18:33:28.883178 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:28.883197 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:28.883240 (+ 43us) server_negotiation.cc:407] Connection header received
0807 18:33:28.884902 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:28.884924 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:28.885250 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:28.885565 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:28.886696 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:28.886701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:28.886717 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:28.886731 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:28.886789 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:28.886809 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:28.887029 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:28.887171 (+ 142us) server_negotiation.cc:299] Negotiation successful
0807 18:33:28.887411 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":339,"threads_started":1}
I20260807 18:33:28.889487 18434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0: Bootstrap starting.
I20260807 18:33:28.891999 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:28.893028 18434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:28.894138 18434 log.cc:824] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:28.897323 18434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0: No bootstrap required, opened a new log
I20260807 18:33:28.898758 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:28.896304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:56565 (local address 127.0.0.1:42809)
0807 18:33:28.896503 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:28.896510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:28.896915 (+ 405us) server_negotiation.cc:407] Connection header received
0807 18:33:28.897035 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:28.897041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:28.897098 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:28.897215 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:28.898111 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:28.898117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:28.898122 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:28.898142 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:28.898187 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:28.898208 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:28.898342 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:28.898392 (+ 50us) server_negotiation.cc:299] Negotiation successful
0807 18:33:28.898451 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 18:33:28.905740 18434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d943141b9274926b089a8d47d9d07a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42809 } }
I20260807 18:33:28.906215 18434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:28.906355 18434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d943141b9274926b089a8d47d9d07a0, State: Initialized, Role: FOLLOWER
I20260807 18:33:28.906998 18434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d943141b9274926b089a8d47d9d07a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42809 } }
I20260807 18:33:28.907320 18434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:28.907480 18434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:28.907680 18434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:28.909960 18434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d943141b9274926b089a8d47d9d07a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42809 } }
I20260807 18:33:28.910462 18434 leader_election.cc:302] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d943141b9274926b089a8d47d9d07a0; no voters:
I20260807 18:33:28.911230 18434 leader_election.cc:288] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:28.911412 18444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:28.912062 18444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d943141b9274926b089a8d47d9d07a0, State: Running, Role: LEADER
I20260807 18:33:28.912686 18444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d943141b9274926b089a8d47d9d07a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42809 } }
I20260807 18:33:28.913661 18434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:28.915984 18446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d943141b9274926b089a8d47d9d07a0. Latest consensus state: current_term: 1 leader_uuid: "4d943141b9274926b089a8d47d9d07a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d943141b9274926b089a8d47d9d07a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42809 } } }
I20260807 18:33:28.915956 18445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d943141b9274926b089a8d47d9d07a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d943141b9274926b089a8d47d9d07a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42809 } } }
I20260807 18:33:28.916538 18446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:28.916600 18445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:28.924304 18449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:28.930295 18449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:28.941612 18449 catalog_manager.cc:1347] Generated new cluster ID: 2ad38d7cbfc44c55990ca12046e2f96c
I20260807 18:33:28.941740 18449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:28.962623 18449 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:28.963084 18449 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:28.970980 18449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d943141b9274926b089a8d47d9d07a0: Generated new TSK 0
I20260807 18:33:28.971865 18449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:28.978411 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18362
W20260807 18:33:28.980223 18429 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.186:36597
W20260807 18:33:28.980898 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:28.981065 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 18362
2026-08-07T18:33:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2012 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:29 dist-test-slave-1xrh krb5kdc[18466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:29 dist-test-slave-1xrh krb5kdc[18466](info): set up 2 sockets
Aug 07 18:33:29 dist-test-slave-1xrh krb5kdc[18466](info): commencing operation
krb5kdc: starting...
W20260807 18:33:30.201267 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.004s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:30 dist-test-slave-1xrh krb5kdc[18466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:33:30.514884 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35189
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:35189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:30.862422 18479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:30.862716 18479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:30.862812 18479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:30.870082 18479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:30.870203 18479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:30.870251 18479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:30.870316 18479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:30.870360 18479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:30.880379 18479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:35189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:30.881665 18479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:30.882941 18479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:30.894989 18485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:30.895151 18486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:30.895190 18488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:30.896040 18479 server_base.cc:1034] running on GCE node
I20260807 18:33:30.896883 18479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:30.898272 18479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:30.899501 18479 hybrid_clock.cc:648] HybridClock initialized: now 1786127610899456 us; error 34 us; skew 500 ppm
I20260807 18:33:30.899838 18479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:30 dist-test-slave-1xrh krb5kdc[18466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127610, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:30.903044 18479 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:33:30.904495 18479 webserver.cc:460] Webserver started at http://127.27.1.254:37423/ using document root <none> and password file <none>
I20260807 18:33:30.905190 18479 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:30.905341 18479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:30.905650 18479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:30.908666 18479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "4c1e9502b4a8471aa5d5322cfa104d10"
format_stamp: "Formatted at 2026-08-07 18:33:30 on dist-test-slave-1xrh"
server_key: "4dcbd83b75ba9ce9420132e77f13acf5"
server_key_iv: "9c37ca1181e90cc7ae76eede7c94f3e5"
server_key_version: "encryptionkey@0"
I20260807 18:33:30.909447 18479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "4c1e9502b4a8471aa5d5322cfa104d10"
format_stamp: "Formatted at 2026-08-07 18:33:30 on dist-test-slave-1xrh"
server_key: "4dcbd83b75ba9ce9420132e77f13acf5"
server_key_iv: "9c37ca1181e90cc7ae76eede7c94f3e5"
server_key_version: "encryptionkey@0"
I20260807 18:33:30.914577 18479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 18:33:30.917992 18495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:30.919258 18479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:33:30.919440 18479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "4c1e9502b4a8471aa5d5322cfa104d10"
format_stamp: "Formatted at 2026-08-07 18:33:30 on dist-test-slave-1xrh"
server_key: "4dcbd83b75ba9ce9420132e77f13acf5"
server_key_iv: "9c37ca1181e90cc7ae76eede7c94f3e5"
server_key_version: "encryptionkey@0"
I20260807 18:33:30.919677 18479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:30.939278 18479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:30.943336 18479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:30.943634 18479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:30.963217 18479 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:35189
I20260807 18:33:30.963213 18548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:35189 every 8 connection(s)
I20260807 18:33:30.964737 18479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:30.966130 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 18479
I20260807 18:33:30.966326 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:30.966945 27655 external_mini_cluster.cc:1428] Setting key 67e1f2115f90b6c3682b18cd553986df
I20260807 18:33:30.969792 18549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 18:33:30 dist-test-slave-1xrh krb5kdc[18466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:33:30.981824 18549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10: Bootstrap starting.
I20260807 18:33:30.985932 18549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:30.987078 18549 log.cc:824] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:30.990640 18549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10: No bootstrap required, opened a new log
I20260807 18:33:30.991732 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:30.968848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38801 (local address 127.27.1.254:35189)
0807 18:33:30.969660 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:30.969675 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:30.969726 (+ 51us) server_negotiation.cc:407] Connection header received
0807 18:33:30.972150 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:30.972177 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:30.972668 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:30.973121 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:30.974571 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:30.976019 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:30.977322 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:30.977983 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:30.978187 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:30.983283 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:30.983296 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:30.983315 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:30.983388 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:30.986947 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:30.988091 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:30.988103 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:30.988112 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:30.988324 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:30.988953 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:30.988957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:30.988959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:30.989498 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:30.989725 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:30.990265 (+ 540us) server_negotiation.cc:299] Negotiation successful
0807 18:33:30.990506 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":344,"threads_started":1}
I20260807 18:33:30.995726 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:31.000610 18549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1e9502b4a8471aa5d5322cfa104d10" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35189 } }
I20260807 18:33:31.001081 18549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:31.001243 18549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1e9502b4a8471aa5d5322cfa104d10, State: Initialized, Role: FOLLOWER
I20260807 18:33:31.001925 18549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1e9502b4a8471aa5d5322cfa104d10" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35189 } }
I20260807 18:33:31.002224 18549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:31.002377 18549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:31.002591 18549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:31.005530 18549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1e9502b4a8471aa5d5322cfa104d10" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35189 } }
I20260807 18:33:31.006057 18549 leader_election.cc:302] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c1e9502b4a8471aa5d5322cfa104d10; no voters:
I20260807 18:33:31.006738 18549 leader_election.cc:288] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:31.006845 18560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:31.008512 18560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [term 1 LEADER]: Becoming Leader. State: Replica: 4c1e9502b4a8471aa5d5322cfa104d10, State: Running, Role: LEADER
I20260807 18:33:31.009344 18549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:31.009202 18560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1e9502b4a8471aa5d5322cfa104d10" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35189 } }
I20260807 18:33:31.012640 18563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c1e9502b4a8471aa5d5322cfa104d10. Latest consensus state: current_term: 1 leader_uuid: "4c1e9502b4a8471aa5d5322cfa104d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1e9502b4a8471aa5d5322cfa104d10" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35189 } } }
I20260807 18:33:31.013010 18563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:31.014657 18561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c1e9502b4a8471aa5d5322cfa104d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1e9502b4a8471aa5d5322cfa104d10" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35189 } } }
I20260807 18:33:31.015041 18561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:31.022594 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:31.001241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36147 (local address 127.27.1.254:35189)
0807 18:33:31.001428 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:31.001434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:31.001924 (+ 490us) server_negotiation.cc:407] Connection header received
0807 18:33:31.002044 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:31.002049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:31.002129 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:31.002256 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:31.003977 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:31.005177 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:31.006621 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:31.007424 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:31.007570 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:31.010903 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:31.010912 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:31.010920 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:31.010975 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:31.016928 (+ 5953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:31.018297 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:31.018305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:31.018312 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:31.018520 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:31.019282 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:31.019290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:31.019295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:31.019736 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:31.021630 (+ 1894us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:31.021867 (+ 237us) server_negotiation.cc:299] Negotiation successful
0807 18:33:31.022099 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 18:33:31.026201 18567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:31.031842 18567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:31.040946 18567 catalog_manager.cc:1347] Generated new cluster ID: 3a9e11bd3e5d4659b58f9b66fdbc3e72
I20260807 18:33:31.041060 18567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:31.058071 18567 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:31.059468 18567 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:31.081173 18567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c1e9502b4a8471aa5d5322cfa104d10: Generated new TSK 0
I20260807 18:33:31.082039 18567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:31.093148 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18479
I20260807 18:33:31.094563 18544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:31.188261 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 18479
W20260807 18:33:31.208535 18464 connection.cc:535] client connection to 127.27.1.254:35189 recv error: Network error: failed to read from TLS socket (remote: 127.27.1.254:35189): Connection reset by peer (error 104)
2026-08-07T18:33:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:31 dist-test-slave-1xrh krb5kdc[18586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:31 dist-test-slave-1xrh krb5kdc[18586](info): set up 2 sockets
Aug 07 18:33:31 dist-test-slave-1xrh krb5kdc[18586](info): commencing operation
krb5kdc: starting...
W20260807 18:33:32.357811 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.002s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:32 dist-test-slave-1xrh krb5kdc[18586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:33:32.666507 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:43709
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:32803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:43709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:33.018186 18599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:33.018532 18599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:33.018725 18599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:33.025992 18599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:33.026119 18599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:33.026225 18599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:33.026324 18599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:33.026400 18599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:33.036116 18599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:32803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:43709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:33.037309 18599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:33.038544 18599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:33.050597 18604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:33.050513 18605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:33.050559 18607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:33.051595 18599 server_base.cc:1034] running on GCE node
I20260807 18:33:33.052325 18599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:33.053694 18599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:33.054915 18599 hybrid_clock.cc:648] HybridClock initialized: now 1786127613054864 us; error 64 us; skew 500 ppm
I20260807 18:33:33.055337 18599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:33 dist-test-slave-1xrh krb5kdc[18586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127613, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:33.058354 18599 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:33:33.059898 18599 webserver.cc:460] Webserver started at http://127.27.1.254:40871/ using document root <none> and password file <none>
I20260807 18:33:33.060570 18599 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:33.060721 18599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:33.061026 18599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:33.064026 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "32bbc60b481742acb340e2f8c474e9d6"
format_stamp: "Formatted at 2026-08-07 18:33:33 on dist-test-slave-1xrh"
server_key: "a9cb70844aade67a20cc64b03646aad8"
server_key_iv: "d4ff8337babc9d80d5b8970020ece7c8"
server_key_version: "encryptionkey@0"
I20260807 18:33:33.064778 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "32bbc60b481742acb340e2f8c474e9d6"
format_stamp: "Formatted at 2026-08-07 18:33:33 on dist-test-slave-1xrh"
server_key: "a9cb70844aade67a20cc64b03646aad8"
server_key_iv: "d4ff8337babc9d80d5b8970020ece7c8"
server_key_version: "encryptionkey@0"
I20260807 18:33:33.069741 18599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 18:33:33.073066 18614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:33.074419 18599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:33:33.074622 18599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "32bbc60b481742acb340e2f8c474e9d6"
format_stamp: "Formatted at 2026-08-07 18:33:33 on dist-test-slave-1xrh"
server_key: "a9cb70844aade67a20cc64b03646aad8"
server_key_iv: "d4ff8337babc9d80d5b8970020ece7c8"
server_key_version: "encryptionkey@0"
I20260807 18:33:33.074822 18599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:33.091995 18599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:33.096136 18599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:33.096431 18599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:33.115186 18599 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:43709
I20260807 18:33:33.115211 18670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:43709 every 8 connection(s)
I20260807 18:33:33.116784 18599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:33.120441 18671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:33.124210 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 18599
I20260807 18:33:33.124387 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:33.124823 27655 external_mini_cluster.cc:1428] Setting key 83e15aae6087cc500ae64e9a1c6c80f2
I20260807 18:33:33.132321 18671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6: Bootstrap starting.
I20260807 18:33:33.135479 18671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:33.136555 18671 log.cc:824] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:33.137717 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:33.126511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52629 (local address 127.27.1.254:43709)
0807 18:33:33.127271 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:33.127282 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:33.127385 (+ 103us) server_negotiation.cc:407] Connection header received
0807 18:33:33.129040 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:33.129060 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:33.129447 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:33.129820 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:33.130365 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:33.131713 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:33.133058 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:33.133706 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:33.133897 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:33.135241 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:33.135249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:33.135262 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:33.135821 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:33.135895 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:33.135918 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:33.136081 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:33.136366 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 18:33:33.136507 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":322,"threads_started":1}
I20260807 18:33:33.139678 18671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6: No bootstrap required, opened a new log
I20260807 18:33:33.141568 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:33.147948 18671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bbc60b481742acb340e2f8c474e9d6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43709 } }
I20260807 18:33:33.148304 18671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:33.148415 18671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32bbc60b481742acb340e2f8c474e9d6, State: Initialized, Role: FOLLOWER
I20260807 18:33:33.149034 18671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bbc60b481742acb340e2f8c474e9d6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43709 } }
I20260807 18:33:33.149277 18671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:33.149426 18671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:33.149611 18671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:33.151979 18671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bbc60b481742acb340e2f8c474e9d6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43709 } }
I20260807 18:33:33.152489 18671 leader_election.cc:302] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32bbc60b481742acb340e2f8c474e9d6; no voters:
I20260807 18:33:33.152911 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:33.146026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56701 (local address 127.27.1.254:43709)
0807 18:33:33.146231 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:33.146238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:33.146742 (+ 504us) server_negotiation.cc:407] Connection header received
0807 18:33:33.146935 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:33.146939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:33.146998 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:33.147077 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:33.147735 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:33.148655 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:33.150133 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:33.150637 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:33.150786 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:33.152026 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:33.152031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:33.152034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:33.152051 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:33.152086 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:33.152101 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:33.152228 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:33.152496 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 18:33:33.152549 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 18:33:33.153281 18671 leader_election.cc:288] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:33.153398 18681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:33.154114 18681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [term 1 LEADER]: Becoming Leader. State: Replica: 32bbc60b481742acb340e2f8c474e9d6, State: Running, Role: LEADER
I20260807 18:33:33.154765 18681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bbc60b481742acb340e2f8c474e9d6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43709 } }
I20260807 18:33:33.156337 18671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:33.157985 18682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32bbc60b481742acb340e2f8c474e9d6. Latest consensus state: current_term: 1 leader_uuid: "32bbc60b481742acb340e2f8c474e9d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bbc60b481742acb340e2f8c474e9d6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43709 } } }
I20260807 18:33:33.157959 18683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32bbc60b481742acb340e2f8c474e9d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bbc60b481742acb340e2f8c474e9d6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 43709 } } }
I20260807 18:33:33.158397 18682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:33.158397 18683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:33.159883 18686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:33.169689 18686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:33.178857 18686 catalog_manager.cc:1347] Generated new cluster ID: 7836bd29e9ee4fa7829398b0a8576b3a
I20260807 18:33:33.178993 18686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:33.198460 18686 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:33.200664 18686 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:33.207542 18686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32bbc60b481742acb340e2f8c474e9d6: Generated new TSK 0
I20260807 18:33:33.208464 18686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:33.215580 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18599
W20260807 18:33:33.216812 18665 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52629
W20260807 18:33:33.217584 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18599: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:33.217724 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 18599
W20260807 18:33:33.239650 18584 connection.cc:535] client connection to 127.27.1.254:43709 recv error: Network error: failed to read from TLS socket (remote: 127.27.1.254:43709): Connection reset by peer (error 104)
2026-08-07T18:33:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2031 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:33 dist-test-slave-1xrh krb5kdc[18703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:33 dist-test-slave-1xrh krb5kdc[18703](info): set up 2 sockets
Aug 07 18:33:33 dist-test-slave-1xrh krb5kdc[18703](info): commencing operation
krb5kdc: starting...
W20260807 18:33:34.437983 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.007s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:34 dist-test-slave-1xrh krb5kdc[18703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:33:34.750268 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:45503
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:35.090801 18717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:35.091078 18717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:35.091264 18717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:35.098315 18717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:35.098419 18717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:35.098508 18717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:35.098613 18717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:35.098691 18717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:35.108371 18717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:45503
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:35.109638 18717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:35.110939 18717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:35.122920 18725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:35.123173 18724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:35.123209 18727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:35.124343 18717 server_base.cc:1034] running on GCE node
I20260807 18:33:35.125142 18717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:35.126504 18717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:35.127753 18717 hybrid_clock.cc:648] HybridClock initialized: now 1786127615127716 us; error 56 us; skew 500 ppm
I20260807 18:33:35.128118 18717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:35 dist-test-slave-1xrh krb5kdc[18703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:35.131368 18717 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:33:35.132833 18717 webserver.cc:460] Webserver started at http://127.27.1.254:43649/ using document root <none> and password file <none>
I20260807 18:33:35.133522 18717 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:35.133668 18717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:35.133980 18717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:35.136956 18717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "59a4871e2dc14be287ba53693b8e69b6"
format_stamp: "Formatted at 2026-08-07 18:33:35 on dist-test-slave-1xrh"
server_key: "b3b7221ea020a6fb36a74b4c724f28f3"
server_key_iv: "de13303282e8d384cb83880f9a370d4c"
server_key_version: "encryptionkey@0"
I20260807 18:33:35.137718 18717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "59a4871e2dc14be287ba53693b8e69b6"
format_stamp: "Formatted at 2026-08-07 18:33:35 on dist-test-slave-1xrh"
server_key: "b3b7221ea020a6fb36a74b4c724f28f3"
server_key_iv: "de13303282e8d384cb83880f9a370d4c"
server_key_version: "encryptionkey@0"
I20260807 18:33:35.143069 18717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 18:33:35.146521 18735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:35.147876 18717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:33:35.148059 18717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "59a4871e2dc14be287ba53693b8e69b6"
format_stamp: "Formatted at 2026-08-07 18:33:35 on dist-test-slave-1xrh"
server_key: "b3b7221ea020a6fb36a74b4c724f28f3"
server_key_iv: "de13303282e8d384cb83880f9a370d4c"
server_key_version: "encryptionkey@0"
I20260807 18:33:35.148257 18717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:35.153951 18717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:35.158624 18717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:35.158910 18717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:35.176647 18717 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:45503
I20260807 18:33:35.176692 18787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:45503 every 8 connection(s)
I20260807 18:33:35.177999 18717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:35.179636 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 18717
I20260807 18:33:35.179819 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:35.180279 27655 external_mini_cluster.cc:1428] Setting key 999d08348a0a8cd11c8d6166586502d9
I20260807 18:33:35.182169 18788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:35.189208 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:35.182179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44321 (local address 127.27.1.254:45503)
0807 18:33:35.183039 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:35.183055 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:35.183112 (+ 57us) server_negotiation.cc:407] Connection header received
0807 18:33:35.185290 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:35.185318 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:35.185756 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:35.186209 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:35.187229 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:35.187239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:35.187257 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:35.187274 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:35.187370 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:35.187397 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:35.187632 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:35.187771 (+ 139us) server_negotiation.cc:299] Negotiation successful
0807 18:33:35.187973 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":349,"threads_started":1}
I20260807 18:33:35.192845 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:35.196985 18788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6: Bootstrap starting.
I20260807 18:33:35.200827 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:35.198300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56209 (local address 127.27.1.254:45503)
0807 18:33:35.198471 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:35.198478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:35.198837 (+ 359us) server_negotiation.cc:407] Connection header received
0807 18:33:35.198974 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:35.198980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:35.199050 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:35.199198 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:35.200114 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:35.200120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:35.200124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:35.200142 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:35.200188 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:35.200208 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:35.200323 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:35.200421 (+ 98us) server_negotiation.cc:299] Negotiation successful
0807 18:33:35.200477 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:33:35.200831 18788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:35.202245 18788 log.cc:824] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:35.205044 18788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6: No bootstrap required, opened a new log
I20260807 18:33:35.213148 18788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a4871e2dc14be287ba53693b8e69b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45503 } }
I20260807 18:33:35.213471 18788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:35.213543 18788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a4871e2dc14be287ba53693b8e69b6, State: Initialized, Role: FOLLOWER
I20260807 18:33:35.214069 18788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a4871e2dc14be287ba53693b8e69b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45503 } }
I20260807 18:33:35.214268 18788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:35.214370 18788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:35.214495 18788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:35.216791 18788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a4871e2dc14be287ba53693b8e69b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45503 } }
I20260807 18:33:35.217217 18788 leader_election.cc:302] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59a4871e2dc14be287ba53693b8e69b6; no voters:
I20260807 18:33:35.217715 18788 leader_election.cc:288] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:35.217890 18799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:35.218570 18799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [term 1 LEADER]: Becoming Leader. State: Replica: 59a4871e2dc14be287ba53693b8e69b6, State: Running, Role: LEADER
I20260807 18:33:35.219061 18799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a4871e2dc14be287ba53693b8e69b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45503 } }
I20260807 18:33:35.219764 18788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:35.221989 18800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59a4871e2dc14be287ba53693b8e69b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a4871e2dc14be287ba53693b8e69b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45503 } } }
I20260807 18:33:35.221793 18801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59a4871e2dc14be287ba53693b8e69b6. Latest consensus state: current_term: 1 leader_uuid: "59a4871e2dc14be287ba53693b8e69b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a4871e2dc14be287ba53693b8e69b6" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45503 } } }
I20260807 18:33:35.222318 18801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:35.222379 18800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:35.223286 18805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:35.228204 18805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:35.237112 18805 catalog_manager.cc:1347] Generated new cluster ID: 9541d735b8944e7d8c9fc9f54a442e24
I20260807 18:33:35.237246 18805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:35.256245 18805 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:35.256748 18805 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:35.267529 18805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59a4871e2dc14be287ba53693b8e69b6: Generated new TSK 0
I20260807 18:33:35.268618 18805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:35.279791 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18717
W20260807 18:33:35.281342 18783 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44321
W20260807 18:33:35.282050 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:35.282191 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 18717
2026-08-07T18:33:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2064 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:35 dist-test-slave-1xrh krb5kdc[18822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:35 dist-test-slave-1xrh krb5kdc[18822](info): set up 2 sockets
Aug 07 18:33:35 dist-test-slave-1xrh krb5kdc[18822](info): commencing operation
krb5kdc: starting...
W20260807 18:33:36.504684 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:36 dist-test-slave-1xrh krb5kdc[18822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:33:36.826838 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:33923
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:33871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:33923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:37.169605 18835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:37.169900 18835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:37.170070 18835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:37.177374 18835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:37.177488 18835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:37.177587 18835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:37.177695 18835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:37.177775 18835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:37.187712 18835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:33871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:33923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:33923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:37.189023 18835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:37.190331 18835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:37.202200 18840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:37.202277 18841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:37.202406 18843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:37.203415 18835 server_base.cc:1034] running on GCE node
I20260807 18:33:37.204195 18835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:37.205622 18835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:37.206871 18835 hybrid_clock.cc:648] HybridClock initialized: now 1786127617206772 us; error 85 us; skew 500 ppm
I20260807 18:33:37.207361 18835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:37 dist-test-slave-1xrh krb5kdc[18822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127617, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:37.210837 18835 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:33:37.212512 18835 webserver.cc:460] Webserver started at http://127.27.1.254:35875/ using document root <none> and password file <none>
I20260807 18:33:37.213250 18835 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:37.213431 18835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:37.213766 18835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:37.216969 18835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "5829c18907744df6bac00d7b46f87f1f"
format_stamp: "Formatted at 2026-08-07 18:33:37 on dist-test-slave-1xrh"
server_key: "acc7234b30d820f49f662025982d89e6"
server_key_iv: "3221641575eb7e9006a7d86d1b175bf8"
server_key_version: "encryptionkey@0"
I20260807 18:33:37.217789 18835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "5829c18907744df6bac00d7b46f87f1f"
format_stamp: "Formatted at 2026-08-07 18:33:37 on dist-test-slave-1xrh"
server_key: "acc7234b30d820f49f662025982d89e6"
server_key_iv: "3221641575eb7e9006a7d86d1b175bf8"
server_key_version: "encryptionkey@0"
I20260807 18:33:37.223114 18835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 18:33:37.226780 18851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:37.228149 18835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:33:37.228355 18835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "5829c18907744df6bac00d7b46f87f1f"
format_stamp: "Formatted at 2026-08-07 18:33:37 on dist-test-slave-1xrh"
server_key: "acc7234b30d820f49f662025982d89e6"
server_key_iv: "3221641575eb7e9006a7d86d1b175bf8"
server_key_version: "encryptionkey@0"
I20260807 18:33:37.228562 18835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:37.243052 18835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:37.247828 18835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:37.248112 18835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:37.265745 18835 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:33923
I20260807 18:33:37.265774 18906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:33923 every 8 connection(s)
I20260807 18:33:37.267110 18835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:37.268348 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 18835
I20260807 18:33:37.268523 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:37.268977 27655 external_mini_cluster.cc:1428] Setting key 86ed09611af20adeb54c0a0fb207a3cc
I20260807 18:33:37.271543 18907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 18:33:37 dist-test-slave-1xrh krb5kdc[18822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:33:37.284403 18907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f: Bootstrap starting.
I20260807 18:33:37.288543 18907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:37.289799 18907 log.cc:824] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:37.292709 18907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f: No bootstrap required, opened a new log
I20260807 18:33:37.293104 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:37.270800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55459 (local address 127.27.1.254:33923)
0807 18:33:37.271652 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:37.271675 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:37.271726 (+ 51us) server_negotiation.cc:407] Connection header received
0807 18:33:37.274089 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:37.274116 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:37.274604 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:37.275045 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:37.276377 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:37.278433 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:37.279768 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:37.280788 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:37.281001 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:37.284928 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:37.284939 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:37.284956 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:37.285004 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:37.288971 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:37.289628 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:37.289637 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:37.289644 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:37.289862 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:37.290286 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:37.290295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:37.290300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:37.290888 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:37.291252 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:37.291692 (+ 440us) server_negotiation.cc:299] Negotiation successful
0807 18:33:37.292089 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":362,"threads_started":1}
I20260807 18:33:37.296831 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:37.301297 18907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829c18907744df6bac00d7b46f87f1f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33923 } }
I20260807 18:33:37.301716 18907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:37.301841 18907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5829c18907744df6bac00d7b46f87f1f, State: Initialized, Role: FOLLOWER
I20260807 18:33:37.302481 18907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829c18907744df6bac00d7b46f87f1f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33923 } }
I20260807 18:33:37.302814 18907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:37.302963 18907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:37.303165 18907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:37.306154 18907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829c18907744df6bac00d7b46f87f1f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33923 } }
I20260807 18:33:37.306705 18907 leader_election.cc:302] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5829c18907744df6bac00d7b46f87f1f; no voters:
I20260807 18:33:37.307392 18907 leader_election.cc:288] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:37.307551 18917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:37.309278 18917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [term 1 LEADER]: Becoming Leader. State: Replica: 5829c18907744df6bac00d7b46f87f1f, State: Running, Role: LEADER
I20260807 18:33:37.309741 18907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:37.310290 18917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829c18907744df6bac00d7b46f87f1f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33923 } }
I20260807 18:33:37.320297 18919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5829c18907744df6bac00d7b46f87f1f. Latest consensus state: current_term: 1 leader_uuid: "5829c18907744df6bac00d7b46f87f1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829c18907744df6bac00d7b46f87f1f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33923 } } }
I20260807 18:33:37.320686 18919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:37.322016 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:37.301174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54743 (local address 127.27.1.254:33923)
0807 18:33:37.301371 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:37.301376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:37.302222 (+ 846us) server_negotiation.cc:407] Connection header received
0807 18:33:37.302367 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:37.302371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:37.302435 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:37.302519 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:37.304545 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:37.305290 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:37.306965 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:37.307832 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:37.308006 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:37.311494 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:37.311500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:37.311504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:37.311540 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:37.315864 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:37.316829 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:37.316834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:37.316837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:37.317025 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:37.317515 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:37.317521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:37.317525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:37.317907 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:37.318199 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:37.321297 (+ 3098us) server_negotiation.cc:299] Negotiation successful
0807 18:33:37.321609 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 18:33:37.322288 18923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:37.322573 18918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5829c18907744df6bac00d7b46f87f1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829c18907744df6bac00d7b46f87f1f" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 33923 } } }
I20260807 18:33:37.322932 18918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:37.327795 18923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:37.341675 18923 catalog_manager.cc:1347] Generated new cluster ID: 8c7688d8423645c5b5a9bb5ec60d37e3
I20260807 18:33:37.341832 18923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:37.352844 18923 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:37.354115 18923 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:37.362109 18923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5829c18907744df6bac00d7b46f87f1f: Generated new TSK 0
I20260807 18:33:37.362946 18923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:37.370234 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18835
I20260807 18:33:37.371349 18902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:37.461501 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 18835
2026-08-07T18:33:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2177 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:37 dist-test-slave-1xrh krb5kdc[18941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:37 dist-test-slave-1xrh krb5kdc[18941](info): set up 2 sockets
Aug 07 18:33:37 dist-test-slave-1xrh krb5kdc[18941](info): commencing operation
krb5kdc: starting...
W20260807 18:33:38.625656 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.001s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:38 dist-test-slave-1xrh krb5kdc[18941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:33:38.942212 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:46021
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:46021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:39.281391 18956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:39.281733 18956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:39.281915 18956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:39.288971 18956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:39.289116 18956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:39.289229 18956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:39.289335 18956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:39.289403 18956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:39.298735 18956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:46021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:46021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:39.300007 18956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:39.301322 18956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:39.313706 18964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:39.313970 18967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:39.313977 18965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:39.316882 18956 server_base.cc:1034] running on GCE node
I20260807 18:33:39.317984 18956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:39.319561 18956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:39.320838 18956 hybrid_clock.cc:648] HybridClock initialized: now 1786127619320722 us; error 138 us; skew 500 ppm
I20260807 18:33:39.321249 18956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:39 dist-test-slave-1xrh krb5kdc[18941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127619, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:39.330577 18956 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:33:39.332304 18956 webserver.cc:460] Webserver started at http://127.27.1.254:33919/ using document root <none> and password file <none>
I20260807 18:33:39.333061 18956 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:39.333216 18956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:39.333554 18956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:39.336627 18956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "12a30b49f5c24f24a31faf4473d989f0"
format_stamp: "Formatted at 2026-08-07 18:33:39 on dist-test-slave-1xrh"
server_key: "70d496a8ed8467dd1a2afb28f27d986e"
server_key_iv: "b650902c63f004f207a5f24b497300ca"
server_key_version: "encryptionkey@0"
I20260807 18:33:39.337406 18956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "12a30b49f5c24f24a31faf4473d989f0"
format_stamp: "Formatted at 2026-08-07 18:33:39 on dist-test-slave-1xrh"
server_key: "70d496a8ed8467dd1a2afb28f27d986e"
server_key_iv: "b650902c63f004f207a5f24b497300ca"
server_key_version: "encryptionkey@0"
I20260807 18:33:39.342621 18956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260807 18:33:39.346199 18974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:39.347528 18956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:33:39.347707 18956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "12a30b49f5c24f24a31faf4473d989f0"
format_stamp: "Formatted at 2026-08-07 18:33:39 on dist-test-slave-1xrh"
server_key: "70d496a8ed8467dd1a2afb28f27d986e"
server_key_iv: "b650902c63f004f207a5f24b497300ca"
server_key_version: "encryptionkey@0"
I20260807 18:33:39.347906 18956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:39.376642 18956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:39.380918 18956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:39.381237 18956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:39.400866 18956 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:46021
I20260807 18:33:39.400949 19027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:46021 every 8 connection(s)
I20260807 18:33:39.402225 18956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:39.405990 19028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:39.411821 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 18956
I20260807 18:33:39.412004 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:39.412613 27655 external_mini_cluster.cc:1428] Setting key 5afebc82c7ae4df73000d102d857b244
I20260807 18:33:39.416985 19028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0: Bootstrap starting.
I20260807 18:33:39.420497 19028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:39.421768 19028 log.cc:824] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:39.425055 19028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0: No bootstrap required, opened a new log
I20260807 18:33:39.426208 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:39.414425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55281 (local address 127.27.1.254:46021)
0807 18:33:39.415323 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:39.415333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:39.415373 (+ 40us) server_negotiation.cc:407] Connection header received
0807 18:33:39.417010 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:39.417032 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:39.417447 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:39.417759 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:39.418367 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:39.419803 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:39.421713 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:39.422725 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:39.422933 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:39.424068 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:39.424075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:39.424088 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:39.424597 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:39.424668 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:39.424692 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:39.424785 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:39.425021 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 18:33:39.425280 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":450,"threads_started":1}
I20260807 18:33:39.429842 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:39.433161 19028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a30b49f5c24f24a31faf4473d989f0" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46021 } }
I20260807 18:33:39.433619 19028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:39.433751 19028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a30b49f5c24f24a31faf4473d989f0, State: Initialized, Role: FOLLOWER
I20260807 18:33:39.434603 19028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a30b49f5c24f24a31faf4473d989f0" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46021 } }
I20260807 18:33:39.434859 19028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:39.435024 19028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:39.435236 19028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:39.437528 19028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a30b49f5c24f24a31faf4473d989f0" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46021 } }
I20260807 18:33:39.438052 19028 leader_election.cc:302] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12a30b49f5c24f24a31faf4473d989f0; no voters:
I20260807 18:33:39.438906 19028 leader_election.cc:288] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:39.439067 19039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:39.440806 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:39.434309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37231 (local address 127.27.1.254:46021)
0807 18:33:39.434488 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:39.434494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:39.434990 (+ 496us) server_negotiation.cc:407] Connection header received
0807 18:33:39.435213 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:39.435227 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:39.435285 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:39.435390 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:39.435961 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:39.436695 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:39.438058 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:39.438561 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:39.438705 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:39.439966 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:39.439972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:39.439977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:39.439996 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:39.440037 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:39.440054 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:39.440115 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:39.440482 (+ 367us) server_negotiation.cc:299] Negotiation successful
0807 18:33:39.440565 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 18:33:39.441557 19028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:39.443440 19039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [term 1 LEADER]: Becoming Leader. State: Replica: 12a30b49f5c24f24a31faf4473d989f0, State: Running, Role: LEADER
I20260807 18:33:39.444185 19039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a30b49f5c24f24a31faf4473d989f0" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46021 } }
I20260807 18:33:39.447527 19041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12a30b49f5c24f24a31faf4473d989f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a30b49f5c24f24a31faf4473d989f0" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46021 } } }
I20260807 18:33:39.447891 19041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:39.448341 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a30b49f5c24f24a31faf4473d989f0. Latest consensus state: current_term: 1 leader_uuid: "12a30b49f5c24f24a31faf4473d989f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a30b49f5c24f24a31faf4473d989f0" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 46021 } } }
I20260807 18:33:39.448639 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:39.450047 19048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:39.456573 19048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:39.465967 19048 catalog_manager.cc:1347] Generated new cluster ID: dd24b7e0a280477298316f7a2f32b5bd
I20260807 18:33:39.466078 19048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:39.487736 19048 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:39.489671 19048 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:39.501080 19048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12a30b49f5c24f24a31faf4473d989f0: Generated new TSK 0
I20260807 18:33:39.502051 19048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:39.508188 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18956
W20260807 18:33:39.509268 19023 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55281
W20260807 18:33:39.510017 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 18956: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:39.510178 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 18956
2026-08-07T18:33:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:39 dist-test-slave-1xrh krb5kdc[19061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:39 dist-test-slave-1xrh krb5kdc[19061](info): set up 2 sockets
Aug 07 18:33:39 dist-test-slave-1xrh krb5kdc[19061](info): commencing operation
krb5kdc: starting...
W20260807 18:33:40.733835 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:40 dist-test-slave-1xrh krb5kdc[19061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:33:41.066897 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34683
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:34683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:41.411654 19074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:41.411967 19074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:41.412150 19074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:41.419238 19074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:41.419380 19074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:41.419483 19074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:41.419596 19074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:41.419684 19074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:41.429391 19074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:34683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:34683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:41.430675 19074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:41.432061 19074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:41.444199 19079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:41.444317 19082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:41.444407 19080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:41.445052 19074 server_base.cc:1034] running on GCE node
I20260807 18:33:41.445806 19074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:41.447220 19074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:41.448482 19074 hybrid_clock.cc:648] HybridClock initialized: now 1786127621448419 us; error 48 us; skew 500 ppm
I20260807 18:33:41.448839 19074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:41 dist-test-slave-1xrh krb5kdc[19061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127621, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:41.452134 19074 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:33:41.453532 19074 webserver.cc:460] Webserver started at http://127.27.1.254:45471/ using document root <none> and password file <none>
I20260807 18:33:41.454206 19074 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:41.454315 19074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:41.454584 19074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:41.457617 19074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788"
format_stamp: "Formatted at 2026-08-07 18:33:41 on dist-test-slave-1xrh"
server_key: "3764c72cdfbfa190dfc6e7ab028f11ce"
server_key_iv: "482cd33f0bc7ddf32fb10ab50ebe9b72"
server_key_version: "encryptionkey@0"
I20260807 18:33:41.458333 19074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788"
format_stamp: "Formatted at 2026-08-07 18:33:41 on dist-test-slave-1xrh"
server_key: "3764c72cdfbfa190dfc6e7ab028f11ce"
server_key_iv: "482cd33f0bc7ddf32fb10ab50ebe9b72"
server_key_version: "encryptionkey@0"
I20260807 18:33:41.463514 19074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 18:33:41.467070 19091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:41.468590 19074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:33:41.468757 19074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788"
format_stamp: "Formatted at 2026-08-07 18:33:41 on dist-test-slave-1xrh"
server_key: "3764c72cdfbfa190dfc6e7ab028f11ce"
server_key_iv: "482cd33f0bc7ddf32fb10ab50ebe9b72"
server_key_version: "encryptionkey@0"
I20260807 18:33:41.468935 19074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:41.474632 19074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:41.478646 19074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:41.478938 19074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:41.497798 19074 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:34683
I20260807 18:33:41.497849 19145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:34683 every 8 connection(s)
I20260807 18:33:41.499257 19074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:41.502925 19146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:41.508543 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 19074
I20260807 18:33:41.508737 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:41.509183 27655 external_mini_cluster.cc:1428] Setting key 1d4eed06f5958bbaf5eccd8128a53be4
I20260807 18:33:41.514691 19146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788: Bootstrap starting.
I20260807 18:33:41.518195 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:41.511026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46183 (local address 127.27.1.254:34683)
0807 18:33:41.512002 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:41.512021 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:41.512075 (+ 54us) server_negotiation.cc:407] Connection header received
0807 18:33:41.514418 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:41.514446 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:41.514863 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:41.515331 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:41.516276 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:41.516290 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:41.516305 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:41.516323 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:41.516422 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:41.516456 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:41.516690 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:41.516818 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 18:33:41.517005 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":391,"threads_started":1}
I20260807 18:33:41.518718 19146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:41.519876 19146 log.cc:824] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:41.521935 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:41.522933 19146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788: No bootstrap required, opened a new log
I20260807 18:33:41.528709 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:41.526325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39429 (local address 127.27.1.254:34683)
0807 18:33:41.526507 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:41.526512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:41.526883 (+ 371us) server_negotiation.cc:407] Connection header received
0807 18:33:41.526997 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:41.527004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:41.527063 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:41.527194 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:41.528102 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:41.528112 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:41.528117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 18:33:41.528135 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:41.528178 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 18:33:41.528199 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:41.528326 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:41.528376 (+ 50us) server_negotiation.cc:299] Negotiation successful
0807 18:33:41.528434 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 18:33:41.531659 19146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34683 } }
I20260807 18:33:41.532023 19146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:33:41.532142 19146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed38e9ab40f4627bc91ed4ef5a0e788, State: Initialized, Role: FOLLOWER
I20260807 18:33:41.532826 19146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34683 } }
I20260807 18:33:41.533145 19146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:33:41.533318 19146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:33:41.533500 19146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:41.535753 19146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34683 } }
I20260807 18:33:41.536234 19146 leader_election.cc:302] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ed38e9ab40f4627bc91ed4ef5a0e788; no voters:
I20260807 18:33:41.536803 19146 leader_election.cc:288] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:33:41.536931 19156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:41.537647 19156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed38e9ab40f4627bc91ed4ef5a0e788, State: Running, Role: LEADER
I20260807 18:33:41.538336 19156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34683 } }
I20260807 18:33:41.539108 19146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:41.541422 19158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ed38e9ab40f4627bc91ed4ef5a0e788. Latest consensus state: current_term: 1 leader_uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34683 } } }
I20260807 18:33:41.542290 19158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:41.541728 19157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed38e9ab40f4627bc91ed4ef5a0e788" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 34683 } } }
I20260807 18:33:41.542994 19157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:41.545049 19165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:41.549369 19165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:41.557523 19165 catalog_manager.cc:1347] Generated new cluster ID: 4edc50abd15d4bf6982c67f7458c087d
I20260807 18:33:41.557655 19165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:41.567291 19165 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:41.567703 19165 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:41.579159 19165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ed38e9ab40f4627bc91ed4ef5a0e788: Generated new TSK 0
I20260807 18:33:41.579944 19165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:41.594214 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 19074
W20260807 18:33:41.595553 19141 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46183
W20260807 18:33:41.596256 27655 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 19074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 18:33:41.596402 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 19074
2026-08-07T18:33:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2087 ms)
[----------] 18 tests from AuthTokenIssuingTest (37691 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:41 dist-test-slave-1xrh krb5kdc[19179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:41 dist-test-slave-1xrh krb5kdc[19179](info): set up 2 sockets
Aug 07 18:33:41 dist-test-slave-1xrh krb5kdc[19179](info): commencing operation
krb5kdc: starting...
W20260807 18:33:42.778261 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:43 dist-test-slave-1xrh krb5kdc[19179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:33:43.126149 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35871
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:35871,127.27.1.253:43787,127.27.1.252:43805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:43.472605 19193 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 18:33:43.472967 19193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:43.473064 19193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:43.473218 19193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:43.480324 19193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:43.480408 19193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:43.480511 19193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:43.480603 19193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:43.480693 19193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:43.490540 19193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:35871,127.27.1.253:43787,127.27.1.252:43805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:35871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:43.491874 19193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:43.493207 19193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:43.504776 19201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:43.504869 19199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:43.505007 19198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:43.506009 19193 server_base.cc:1034] running on GCE node
I20260807 18:33:43.506695 19193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:43.508105 19193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:43.509354 19193 hybrid_clock.cc:648] HybridClock initialized: now 1786127623509296 us; error 33 us; skew 500 ppm
I20260807 18:33:43.509678 19193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:43 dist-test-slave-1xrh krb5kdc[19179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127623, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:43.513000 19193 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:33:43.514497 19193 webserver.cc:460] Webserver started at http://127.27.1.254:43403/ using document root <none> and password file <none>
I20260807 18:33:43.515262 19193 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:43.515399 19193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:43.515731 19193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:43.518661 19193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "980f7bb423eb466497a8a7a9fe9a1aad"
format_stamp: "Formatted at 2026-08-07 18:33:43 on dist-test-slave-1xrh"
server_key: "f73f57912867eb73be5ece6f423412e6"
server_key_iv: "b16f678703226eed6a1ebbe3d07122cb"
server_key_version: "encryptionkey@0"
I20260807 18:33:43.519529 19193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "980f7bb423eb466497a8a7a9fe9a1aad"
format_stamp: "Formatted at 2026-08-07 18:33:43 on dist-test-slave-1xrh"
server_key: "f73f57912867eb73be5ece6f423412e6"
server_key_iv: "b16f678703226eed6a1ebbe3d07122cb"
server_key_version: "encryptionkey@0"
I20260807 18:33:43.524605 19193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 18:33:43.527959 19208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:43.529281 19193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 18:33:43.529470 19193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "980f7bb423eb466497a8a7a9fe9a1aad"
format_stamp: "Formatted at 2026-08-07 18:33:43 on dist-test-slave-1xrh"
server_key: "f73f57912867eb73be5ece6f423412e6"
server_key_iv: "b16f678703226eed6a1ebbe3d07122cb"
server_key_version: "encryptionkey@0"
I20260807 18:33:43.529691 19193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:43.552417 19193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:43.556967 19193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:43.557251 19193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:43.576102 19193 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:35871
I20260807 18:33:43.576136 19263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:35871 every 8 connection(s)
I20260807 18:33:43.577392 19193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:43.580940 19264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:43.586179 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 19193
I20260807 18:33:43.586395 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:43.587262 27655 external_mini_cluster.cc:1428] Setting key dd157dbb024dc1599474e445681e38cc
I20260807 18:33:43.588207 19264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:33:43 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127623, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:33:43.615027 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.592140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35871 (local address 127.0.0.1:37582)
0807 18:33:43.593115 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.593165 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:43.596179 (+ 3014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:43.597002 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:43.597016 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:43.597931 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:43.598410 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:43.598437 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:43.600161 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:43.600169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:43.602067 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:43.602086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:43.603201 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:43.603209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:43.603345 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:43.604062 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:43.604085 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:43.606336 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:43.610804 (+ 4468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:43.610815 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:43.610826 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:43.611353 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:43.611846 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:43.611850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:43.611852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:43.612058 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:43.613134 (+ 1076us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:43.613140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:43.613334 (+ 194us) client_negotiation.cc:769] Sending connection context
0807 18:33:43.613565 (+ 231us) client_negotiation.cc:240] Negotiation successful
0807 18:33:43.613969 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":417,"threads_started":1}
I20260807 18:33:43.615088 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.592140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.27.1.254:35871)
0807 18:33:43.594049 (+ 1909us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:43.594055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:43.594098 (+ 43us) server_negotiation.cc:407] Connection header received
0807 18:33:43.596556 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:43.596561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:43.596763 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:43.596945 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:43.598630 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:43.599985 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:43.602223 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:43.602887 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:43.603083 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:43.606591 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:43.606604 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:43.606622 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:43.606664 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:43.610654 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:43.611495 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:43.611507 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:43.611513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:43.611703 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:43.612188 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:43.612195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:43.612201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:43.612983 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:43.613191 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:43.613770 (+ 579us) server_negotiation.cc:299] Negotiation successful
0807 18:33:43.614197 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":1727,"thread_start_us":1472,"threads_started":1}
I20260807 18:33:43.618197 19264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
I20260807 18:33:43.619288 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.618833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:48944)
0807 18:33:43.619027 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.619173 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260807 18:33:43.619519 19211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
I20260807 18:33:43.621155 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.620789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:48960)
0807 18:33:43.620935 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.621026 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 18:33:43.621366 19264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:43787: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.1.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab.
Entry for principal kudu/127.27.1.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab.
I20260807 18:33:43.664824 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.253:43787
--webserver_interface=127.27.1.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab
--principal=kudu/127.27.1.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:35871,127.27.1.253:43787,127.27.1.252:43805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:33:43.670686 19264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } attempt: 1
I20260807 18:33:43.671854 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.671412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:48966)
0807 18:33:43.671612 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.671706 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 18:33:43.673014 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.672579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:48970)
0807 18:33:43.672802 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.672882 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 18:33:43.673203 19264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:43787: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
I20260807 18:33:43.741457 19264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } attempt: 2
I20260807 18:33:43.742954 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.742401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:48982)
0807 18:33:43.742668 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.742806 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 18:33:43.744079 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.743680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:48998)
0807 18:33:43.743846 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.743938 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 18:33:43.744251 19264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:43787: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
I20260807 18:33:43.835489 19264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } attempt: 3
I20260807 18:33:43.836738 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.836147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:49004)
0807 18:33:43.836406 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.836533 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260807 18:33:43.837865 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.837435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:49006)
0807 18:33:43.837599 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.837690 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 18:33:43.838073 19264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:43787: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
I20260807 18:33:43.981364 19264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } attempt: 4
I20260807 18:33:43.982784 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.982112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:49012)
0807 18:33:43.982452 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.982594 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260807 18:33:43.983955 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:43.983498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:49024)
0807 18:33:43.983669 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:43.983778 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 18:33:43.984169 19264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:43787: Network error: Client connection negotiation failed: client connection to 127.27.1.253:43787: connect: Connection refused (error 111)
W20260807 18:33:44.008858 19271 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 18:33:44.009177 19271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:44.009243 19271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:44.009317 19271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:44.016438 19271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:44.016528 19271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:44.016585 19271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:44.016637 19271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:44.016703 19271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:44.026216 19271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:35871,127.27.1.253:43787,127.27.1.252:43805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab
--principal=kudu/127.27.1.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.253:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.1.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:44.027503 19271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:44.028765 19271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:44.040881 19276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:44.041687 19277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:44.042374 19279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:44.042483 19271 server_base.cc:1034] running on GCE node
I20260807 18:33:44.043360 19271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:44.044744 19271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:44.046005 19271 hybrid_clock.cc:648] HybridClock initialized: now 1786127624045947 us; error 48 us; skew 500 ppm
I20260807 18:33:44.046345 19271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:44.049535 19271 init.cc:377] Logged in from keytab as kudu/127.27.1.253@KRBTEST.COM (short username kudu)
I20260807 18:33:44.051082 19271 webserver.cc:460] Webserver started at http://127.27.1.253:39249/ using document root <none> and password file <none>
I20260807 18:33:44.051790 19271 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:44.051931 19271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:44.052263 19271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:44.055187 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/data/instance:
uuid: "03c9391be69644ea98cdbf8e1bfc937d"
format_stamp: "Formatted at 2026-08-07 18:33:44 on dist-test-slave-1xrh"
server_key: "16f2bb58492cbf76ecfcf19bde6c2e18"
server_key_iv: "79b561bd1f19c6c7aa069631ca34dd98"
server_key_version: "encryptionkey@0"
I20260807 18:33:44.055997 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/wal/instance:
uuid: "03c9391be69644ea98cdbf8e1bfc937d"
format_stamp: "Formatted at 2026-08-07 18:33:44 on dist-test-slave-1xrh"
server_key: "16f2bb58492cbf76ecfcf19bde6c2e18"
server_key_iv: "79b561bd1f19c6c7aa069631ca34dd98"
server_key_version: "encryptionkey@0"
I20260807 18:33:44.061312 19271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260807 18:33:44.065245 19286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:44.066519 19271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 18:33:44.066705 19271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/wal
uuid: "03c9391be69644ea98cdbf8e1bfc937d"
format_stamp: "Formatted at 2026-08-07 18:33:44 on dist-test-slave-1xrh"
server_key: "16f2bb58492cbf76ecfcf19bde6c2e18"
server_key_iv: "79b561bd1f19c6c7aa069631ca34dd98"
server_key_version: "encryptionkey@0"
I20260807 18:33:44.066936 19271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:44.085803 19271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:44.089710 19271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:44.089999 19271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:44.108893 19271 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.253:43787
I20260807 18:33:44.108901 19339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.253:43787 every 8 connection(s)
I20260807 18:33:44.110262 19271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
I20260807 18:33:44.113919 19340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:44.116026 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 19271
I20260807 18:33:44.116230 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-1/wal/instance
I20260807 18:33:44.116781 27655 external_mini_cluster.cc:1428] Setting key 3cd891726306955cc6d6dbb1f4460432
I20260807 18:33:44.123363 19340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:33:44.147818 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.126726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.27.1.254:35871)
0807 18:33:44.127310 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.127317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.128306 (+ 989us) server_negotiation.cc:407] Connection header received
0807 18:33:44.131219 (+ 2913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.131223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.131296 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.131423 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.133090 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.134450 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.136038 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.136786 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.136962 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.140409 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.140417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.140425 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.140476 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.144419 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.145150 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.145158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.145164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.145344 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.145764 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.145771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.145777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.146191 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.146345 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.147044 (+ 699us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.147331 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":283,"threads_started":1}
I20260807 18:33:44.148476 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.127261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35871 (local address 127.0.0.1:37590)
0807 18:33:44.128119 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.128167 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:44.130861 (+ 2694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:44.131693 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:44.131706 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:44.132318 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:44.132864 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.132880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.134627 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.134631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.135885 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.135898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.137035 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.137039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.137222 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.137917 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:44.137940 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:44.140168 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:44.144554 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.144561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.144572 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.145011 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.145463 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.145466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.145481 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.145639 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.146315 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:44.146321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:44.146655 (+ 334us) client_negotiation.cc:769] Sending connection context
0807 18:33:44.146882 (+ 227us) client_negotiation.cc:240] Negotiation successful
0807 18:33:44.147314 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":323,"threads_started":1}
I20260807 18:33:44.150266 19340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.1.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.252@KRBTEST.COM" created.
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:33:44.166340 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.150942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:49032)
0807 18:33:44.151157 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.151177 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:44.151274 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:44.152257 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:44.152265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:44.152747 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:44.152898 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.152907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.154204 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.154207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.155253 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.155271 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.156282 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.156294 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.156509 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.157115 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:44.157143 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:44.159650 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:44.163996 (+ 4346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.164000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.164003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.164353 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.164776 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.164780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.164782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.164905 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.165653 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:44.165660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:44.165764 (+ 104us) client_negotiation.cc:769] Sending connection context
0807 18:33:44.165863 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 18:33:44.166078 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 18:33:44.166868 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.151240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.27.1.253:43787)
0807 18:33:44.151848 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.151853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.151878 (+ 25us) server_negotiation.cc:407] Connection header received
0807 18:33:44.151917 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.151922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.152081 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.152186 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.153057 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.154074 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.155407 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.156120 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.156221 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.159880 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.159896 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.159914 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.159962 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.163844 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.164498 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.164509 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.164515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.164666 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.165022 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.165029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.165035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.165508 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.165742 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.166099 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.166376 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":311,"threads_started":1}
I20260807 18:33:44.169222 19340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } has no permanent_uuid. Determining permanent_uuid...
I20260807 18:33:44.170506 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.170021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51088)
0807 18:33:44.170238 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.170372 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 18:33:44.170838 19287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
I20260807 18:33:44.172473 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.172141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51092)
0807 18:33:44.172294 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.172378 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 18:33:44.172709 19340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:43805: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab.
Entry for principal kudu/127.27.1.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab.
I20260807 18:33:44.194743 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.252:43805
--webserver_interface=127.27.1.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab
--principal=kudu/127.27.1.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:35871,127.27.1.253:43787,127.27.1.252:43805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:33:44.222002 19340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } attempt: 1
I20260807 18:33:44.223414 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.222738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51102)
0807 18:33:44.223023 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.223186 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 18:33:44.224649 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.224164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51118)
0807 18:33:44.224348 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.224482 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 18:33:44.224874 19340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:43805: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
I20260807 18:33:44.283430 19264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } attempt: 5
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127623, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:33:44.293103 19340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } attempt: 2
I20260807 18:33:44.294080 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.293613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51126)
0807 18:33:44.293787 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.293871 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 18:33:44.295107 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.294746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51138)
0807 18:33:44.294891 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.294975 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 18:33:44.295433 19340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:43805: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
I20260807 18:33:44.298125 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.284234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:49040)
0807 18:33:44.284461 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.284492 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:44.284608 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:44.284983 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:44.284986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:44.285410 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:44.285592 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.285601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.287047 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.287050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.287928 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.287940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.288901 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.288912 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.289071 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.289719 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:44.289737 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:44.291783 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:44.295939 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.295942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.295945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.296311 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.296696 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.296699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.296702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.296829 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.297431 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:44.297434 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:44.297567 (+ 133us) client_negotiation.cc:769] Sending connection context
0807 18:33:44.297702 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 18:33:44.297894 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 18:33:44.298400 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.284361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.27.1.253:43787)
0807 18:33:44.284601 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.284607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.284633 (+ 26us) server_negotiation.cc:407] Connection header received
0807 18:33:44.284700 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.284706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.284807 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.284919 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.285756 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.286908 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.288082 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.288747 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.288884 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.291981 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.291988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.291993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.292033 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.295815 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.296431 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.296437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.296440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.296585 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.296936 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.296941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.296944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.297291 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.297452 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.297828 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.298055 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260807 18:33:44.299467 19264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } has no permanent_uuid. Determining permanent_uuid...
I20260807 18:33:44.300599 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.300151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51150)
0807 18:33:44.300326 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.300434 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 18:33:44.301630 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.301255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51158)
0807 18:33:44.301413 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.301497 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 18:33:44.301841 19264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:43805: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
I20260807 18:33:44.353085 19264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } attempt: 1
I20260807 18:33:44.354408 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.353826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51170)
0807 18:33:44.354055 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.354169 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 18:33:44.355507 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.355091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51186)
0807 18:33:44.355276 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.355368 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260807 18:33:44.355718 19264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:43805: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
I20260807 18:33:44.386718 19340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } attempt: 3
I20260807 18:33:44.388069 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.387561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51196)
0807 18:33:44.387805 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.387904 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 18:33:44.389174 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.388759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51200)
0807 18:33:44.388972 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.389056 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260807 18:33:44.389389 19340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:43805: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
I20260807 18:33:44.423950 19264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } attempt: 2
I20260807 18:33:44.425379 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.424696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51210)
0807 18:33:44.424955 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.425111 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 18:33:44.426523 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.426107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51220)
0807 18:33:44.426283 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.426380 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 18:33:44.426769 19264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:43805: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
I20260807 18:33:44.532610 19340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } attempt: 4
I20260807 18:33:44.533010 19264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } attempt: 3
I20260807 18:33:44.534024 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.533390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51234)
0807 18:33:44.533724 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.533835 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 18:33:44.534389 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.533646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51242)
0807 18:33:44.533823 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.533923 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 18:33:44.535064 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.534758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51244)
0807 18:33:44.534887 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.534966 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260807 18:33:44.535293 19340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:43805: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
I20260807 18:33:44.535684 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.535055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51260)
0807 18:33:44.535446 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.535548 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
W20260807 18:33:44.535945 19264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:43805: Network error: Client connection negotiation failed: client connection to 127.27.1.252:43805: connect: Connection refused (error 111)
W20260807 18:33:44.544968 19349 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 18:33:44.545248 19349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:44.545318 19349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:44.545447 19349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:44.552852 19349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:44.552932 19349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:44.552990 19349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:44.553041 19349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:44.553107 19349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:44.563611 19349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:35871,127.27.1.253:43787,127.27.1.252:43805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab
--principal=kudu/127.27.1.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.252:43805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.1.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:44.564906 19349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:44.566244 19349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:44.578296 19355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:44.578811 19354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:44.578845 19357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:44.579452 19349 server_base.cc:1034] running on GCE node
I20260807 18:33:44.580222 19349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:44.581571 19349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:44.582819 19349 hybrid_clock.cc:648] HybridClock initialized: now 1786127624582749 us; error 48 us; skew 500 ppm
I20260807 18:33:44.583206 19349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:44.593047 19349 init.cc:377] Logged in from keytab as kudu/127.27.1.252@KRBTEST.COM (short username kudu)
I20260807 18:33:44.594564 19349 webserver.cc:460] Webserver started at http://127.27.1.252:33665/ using document root <none> and password file <none>
I20260807 18:33:44.595329 19349 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:44.595464 19349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:44.595794 19349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:44.598814 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/data/instance:
uuid: "8697b2c867d24292b6f42f4c2559e458"
format_stamp: "Formatted at 2026-08-07 18:33:44 on dist-test-slave-1xrh"
server_key: "c7abcf3094b14bd4cfc497a953c0ffd8"
server_key_iv: "6960d7c323f18544872e551c51a9765e"
server_key_version: "encryptionkey@0"
I20260807 18:33:44.599661 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/wal/instance:
uuid: "8697b2c867d24292b6f42f4c2559e458"
format_stamp: "Formatted at 2026-08-07 18:33:44 on dist-test-slave-1xrh"
server_key: "c7abcf3094b14bd4cfc497a953c0ffd8"
server_key_iv: "6960d7c323f18544872e551c51a9765e"
server_key_version: "encryptionkey@0"
I20260807 18:33:44.604732 19349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:33:44.608232 19366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:44.609544 19349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:33:44.609731 19349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/wal
uuid: "8697b2c867d24292b6f42f4c2559e458"
format_stamp: "Formatted at 2026-08-07 18:33:44 on dist-test-slave-1xrh"
server_key: "c7abcf3094b14bd4cfc497a953c0ffd8"
server_key_iv: "6960d7c323f18544872e551c51a9765e"
server_key_version: "encryptionkey@0"
I20260807 18:33:44.609948 19349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:44.646518 19349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:44.650722 19349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:44.651012 19349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:44.670092 19349 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.252:43805
I20260807 18:33:44.670149 19418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.252:43805 every 8 connection(s)
I20260807 18:33:44.671614 19349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
I20260807 18:33:44.675310 19419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:44.677737 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 19349
I20260807 18:33:44.677919 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786127255008662-27655-0/minicluster-data/master-2/wal/instance
I20260807 18:33:44.678395 27655 external_mini_cluster.cc:1428] Setting key ed81e51abe9b61fee5eebd8379ead5f2
I20260807 18:33:44.683615 19419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:33:44.703043 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.686414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.27.1.254:35871)
0807 18:33:44.686947 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.686952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.687460 (+ 508us) server_negotiation.cc:407] Connection header received
0807 18:33:44.687708 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.687713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.687794 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.687912 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.689374 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.690284 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.692116 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.692523 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.692620 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.696348 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.696352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.696355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.696414 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.699755 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.700617 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.700622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.700626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.700776 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.701260 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.701265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.701268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.701637 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.701809 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.702431 (+ 622us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.702646 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":253,"threads_started":1}
I20260807 18:33:44.703482 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.686484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:35871 (local address 127.0.0.1:37598)
0807 18:33:44.687313 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.687355 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:44.687549 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:44.687990 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:44.687997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:44.688862 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:44.689202 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.689214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.690450 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.690454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.691939 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.691958 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.692773 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.692783 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.693002 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.693637 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:44.693678 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:44.696152 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:44.699884 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.699898 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.699937 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.700465 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.700893 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.700896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.700898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.701111 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.701805 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:44.701813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:44.702027 (+ 214us) client_negotiation.cc:769] Sending connection context
0807 18:33:44.702204 (+ 177us) client_negotiation.cc:240] Negotiation successful
0807 18:33:44.702517 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":409,"threads_started":1}
I20260807 18:33:44.703531 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.680353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.27.1.252:43805)
0807 18:33:44.681151 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.681166 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.681227 (+ 61us) server_negotiation.cc:407] Connection header received
0807 18:33:44.683602 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.683631 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.684155 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.684618 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.686101 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.688028 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.689805 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.690745 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.690878 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.695707 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.695715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.695726 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.695758 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.699315 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.699999 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.700008 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.700010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.700163 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.700620 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.700624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.700627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.701281 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.701574 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.701944 (+ 370us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.702322 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":334,"threads_started":1}
I20260807 18:33:44.705045 19419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:33:44.713155 19264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } attempt: 4
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:33:44.720227 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.705692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:43787 (local address 127.0.0.1:49054)
0807 18:33:44.705855 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.705875 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:44.705971 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:44.706438 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:44.706443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:44.706934 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:44.707118 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.707182 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.708191 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.708199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.709190 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.709204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.709899 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.709905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.710091 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.710730 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:44.710748 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:44.713106 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:44.717492 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.717498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.717501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.718002 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.718396 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.718401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.718404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.718554 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.719203 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:44.719214 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:44.719381 (+ 167us) client_negotiation.cc:769] Sending connection context
0807 18:33:44.719560 (+ 179us) client_negotiation.cc:240] Negotiation successful
0807 18:33:44.719857 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 18:33:44.720479 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.705949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.27.1.253:43787)
0807 18:33:44.706174 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.706179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.706194 (+ 15us) server_negotiation.cc:407] Connection header received
0807 18:33:44.706241 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.706245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.706302 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.706405 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.707330 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.708042 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.709338 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.709752 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.709845 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.713307 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.713316 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.713323 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.713366 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.717360 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.718132 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.718138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.718141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.718281 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.718670 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.718676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.718680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.719051 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.719251 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.719678 (+ 427us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.719900 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260807 18:33:44.721585 19419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127623, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
I20260807 18:33:44.724277 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.706621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.27.1.254:35871)
0807 18:33:44.706770 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.706774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.706870 (+ 96us) server_negotiation.cc:407] Connection header received
0807 18:33:44.707038 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.707041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.707101 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.707233 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.708388 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.709036 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.710561 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.711088 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.711278 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.716795 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.716801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.716805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.716845 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.720925 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.721722 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.721729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.721733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.721875 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.722333 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.722340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.722344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.722715 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.723024 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.723589 (+ 565us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.723841 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 18:33:44.731297 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.714390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.27.1.252:43805)
0807 18:33:44.714556 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.714562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.714583 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:33:44.714638 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.714643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.714717 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.715753 (+ 1036us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.715916 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.716947 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.718965 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.719595 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.719701 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.723984 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.723994 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.723999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.724035 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.728197 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.729025 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.729031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.729034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.729186 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.729632 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.729637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.729641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.730036 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.730190 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.730688 (+ 498us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.730929 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 18:33:44.731302 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.713779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51268)
0807 18:33:44.714008 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.714048 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:44.714153 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:44.714888 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:44.714895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:44.715388 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:44.715586 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.715601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.717107 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.717113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.718811 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.718830 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.719883 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.719889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.720120 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.720760 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:44.720785 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:44.723731 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:44.728334 (+ 4603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.728340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.728344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.728864 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.729322 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.729327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.729331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.729523 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.730192 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:44.730199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:44.730387 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 18:33:44.730554 (+ 167us) client_negotiation.cc:240] Negotiation successful
0807 18:33:44.730875 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:33:44.738632 19264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad: Bootstrap starting.
I20260807 18:33:44.738993 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.722314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51282)
0807 18:33:44.722495 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.722515 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:44.722617 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:44.723550 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:44.723556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:44.724043 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:44.724240 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.724250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.726636 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.726641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.728186 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.728196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.728898 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.728903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.729105 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.729731 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:44.729754 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:44.732320 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:44.736199 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.736205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.736210 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.736722 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.737177 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.737183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.737187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.737336 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.737998 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:44.738006 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:44.738155 (+ 149us) client_negotiation.cc:769] Sending connection context
0807 18:33:44.738341 (+ 186us) client_negotiation.cc:240] Negotiation successful
0807 18:33:44.738656 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 18:33:44.739979 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.722558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51282 (local address 127.27.1.252:43805)
0807 18:33:44.723198 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.723204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.723225 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:33:44.723270 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.723274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.723342 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.724827 (+ 1485us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.725109 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.726469 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.728340 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.728750 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.728878 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.732536 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.732541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.732545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.732570 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.736065 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.736865 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.736869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.736872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.737015 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.737488 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.737492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.737495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.737866 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.739271 (+ 1405us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.739407 (+ 136us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.739626 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":297,"threads_started":1}
I20260807 18:33:44.742287 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.726235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.27.1.253:43787)
0807 18:33:44.726439 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.726446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.726466 (+ 20us) server_negotiation.cc:407] Connection header received
0807 18:33:44.726528 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.726533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.726596 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.726702 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.728059 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.729056 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.730767 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.731533 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.731644 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.736558 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.736562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.736566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.736597 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.739437 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.740077 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.740084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.740088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.740230 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.740674 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.740679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.740682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.741042 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.741283 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.741714 (+ 431us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.741950 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 18:33:44.743181 19264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:44.744593 19264 log.cc:824] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:44.745872 19419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458: Bootstrap starting.
I20260807 18:33:44.747246 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:44.748224 19264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad: No bootstrap required, opened a new log
I20260807 18:33:44.749797 19419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:44.751088 19419 log.cc:824] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:44.749963 19237 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "980f7bb423eb466497a8a7a9fe9a1aad"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37614
I20260807 18:33:44.754304 19419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458: No bootstrap required, opened a new log
I20260807 18:33:44.759330 19264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } }
I20260807 18:33:44.759723 19264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f7bb423eb466497a8a7a9fe9a1aad, State: Initialized, Role: FOLLOWER
I20260807 18:33:44.760282 19264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } }
I20260807 18:33:44.761332 19428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } } }
I20260807 18:33:44.761757 19428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:44.762575 19264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:44.762641 19237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 18:33:44.762885 19237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:44.765237 19419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } }
I20260807 18:33:44.765679 19419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8697b2c867d24292b6f42f4c2559e458, State: Initialized, Role: FOLLOWER
I20260807 18:33:44.766176 19419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } }
I20260807 18:33:44.766538 19237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } }
I20260807 18:33:44.769244 19237 leader_election.cc:288] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [CANDIDATE]: Term 1 election: Requested vote from peers 03c9391be69644ea98cdbf8e1bfc937d (127.27.1.253:43787), 8697b2c867d24292b6f42f4c2559e458 (127.27.1.252:43805)
I20260807 18:33:44.773321 19436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } } }
I20260807 18:33:44.773757 19436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:44.774942 19419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:44.775004 19393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8697b2c867d24292b6f42f4c2559e458"
I20260807 18:33:44.775664 19393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:44.779573 19393 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f7bb423eb466497a8a7a9fe9a1aad in term 1.
I20260807 18:33:44.780503 19209 leader_election.cc:302] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8697b2c867d24292b6f42f4c2559e458, 980f7bb423eb466497a8a7a9fe9a1aad; no voters:
I20260807 18:33:44.782095 19428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [term 1 FOLLOWER]: Leader election won for term 1
W20260807 18:33:44.784876 19440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 18:33:44.787321 19440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260807 18:33:44.810313 19214 tablet.cc:2369] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:33:44.813548 19428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [term 1 LEADER]: Becoming Leader. State: Replica: 980f7bb423eb466497a8a7a9fe9a1aad, State: Running, Role: LEADER
I20260807 18:33:44.814476 19428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } }
I20260807 18:33:44.829409 19460 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 18:33:44.834506 19340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } attempt: 5
Aug 07 18:33:44 dist-test-slave-1xrh krb5kdc[19179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
I20260807 18:33:44.843992 19449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980f7bb423eb466497a8a7a9fe9a1aad. Latest consensus state: current_term: 1 leader_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } } }
I20260807 18:33:44.829286 19193 master_runner.cc:186] Error getting master registration for 127.27.1.253:43787: OK, instance_id { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" instance_seqno: 1786127624103316 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 18:33:44.846220 19193 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 18:33:44.839991 19349 master_runner.cc:186] Error getting master registration for 127.27.1.253:43787: OK, instance_id { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" instance_seqno: 1786127624103316 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 18:33:44.851055 19349 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 18:33:44.845877 19449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:44.852486 19462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:44.854442 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.835485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:43805 (local address 127.0.0.1:51308)
0807 18:33:44.835651 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:44.835669 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:44.835773 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:44.836562 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:44.836567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:44.837085 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:44.837303 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.837318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.838767 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.838772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.840173 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:44.840187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.841516 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.841522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:44.841744 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.842453 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:44.842477 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:44.845440 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:44.851223 (+ 5783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.851229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.851233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.851842 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.852342 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:44.852347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:44.852350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:44.852553 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:44.853420 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:44.853426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:44.853587 (+ 161us) client_negotiation.cc:769] Sending connection context
0807 18:33:44.853742 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 18:33:44.854046 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 18:33:44.855492 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.835688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.27.1.252:43805)
0807 18:33:44.836157 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.836162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.836179 (+ 17us) server_negotiation.cc:407] Connection header received
0807 18:33:44.836230 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.836235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.836299 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.836390 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.837512 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.838565 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.840344 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.841110 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.841263 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.845676 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.845684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.845691 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.845736 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.850776 (+ 5040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.852024 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.852028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.852031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.852194 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.852757 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.852765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.852769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.853195 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.853350 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.854899 (+ 1549us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.855143 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336}
I20260807 18:33:44.856598 19462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:33:44.862022 19340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d: Bootstrap starting.
I20260807 18:33:44.865643 19393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [term 1 FOLLOWER]: Refusing update from remote peer 980f7bb423eb466497a8a7a9fe9a1aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:33:44.866175 19340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:44.866672 19449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:33:44.867558 19340 log.cc:824] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:44.871601 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.796995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.27.1.254:35871)
0807 18:33:44.797190 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.797197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.818951 (+ 21754us) server_negotiation.cc:407] Connection header received
0807 18:33:44.820203 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.820212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.820325 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.820434 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.831099 (+ 10665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.832421 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.842350 (+ 9929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.844416 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.844539 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.857434 (+ 12895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.857443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.857451 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.857513 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.861476 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.867667 (+ 6191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.867673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.867676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.867844 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.869464 (+ 1620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.869469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.869472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.869925 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.870155 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.870933 (+ 778us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.871197 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 18:33:44.871634 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.826754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.27.1.253:43787)
0807 18:33:44.826977 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.826984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.833983 (+ 6999us) server_negotiation.cc:407] Connection header received
0807 18:33:44.834844 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.834851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.834947 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.835027 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.848645 (+ 13618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.849702 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.851803 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.852496 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.852600 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.856317 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.856326 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.856332 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.856372 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.863260 (+ 6888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.868782 (+ 5522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.868790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.868794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.868962 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.869461 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.869466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.869469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.869903 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.870155 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.870883 (+ 728us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.871197 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 18:33:44.876407 19436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980f7bb423eb466497a8a7a9fe9a1aad. Latest consensus state: current_term: 1 leader_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } } }
I20260807 18:33:44.876646 19340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d: No bootstrap required, opened a new log
I20260807 18:33:44.876729 19436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:44.878791 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:44.799399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51294 (local address 127.27.1.252:43805)
0807 18:33:44.799645 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:44.799654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:44.828134 (+ 28480us) server_negotiation.cc:407] Connection header received
0807 18:33:44.829527 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:44.829533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:44.829664 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:44.829763 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:44.838451 (+ 8688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.839514 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.842824 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:44.843564 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:44.843674 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:44.868396 (+ 24722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:44.868403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:44.868419 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:44.868462 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:44.872630 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.876241 (+ 3611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.876246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.876249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.876436 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:44.876934 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:44.876939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:44.876942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:44.877357 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:44.877598 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:44.878185 (+ 587us) server_negotiation.cc:299] Negotiation successful
0807 18:33:44.878442 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:33:44.882251 19436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } } }
I20260807 18:33:44.882475 19436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:44.887459 19464 mvcc.cc:205] Tried to move back new op lower bound from 7315978751439011840 to 7315978751248445440. Current Snapshot: MvccSnapshot[applied={T|T < 7315978751439011840}]
I20260807 18:33:44.889816 19340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } }
I20260807 18:33:44.890246 19340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c9391be69644ea98cdbf8e1bfc937d, State: Initialized, Role: FOLLOWER
I20260807 18:33:44.890792 19340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } }
I20260807 18:33:44.892181 19470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } } }
I20260807 18:33:44.893069 19470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:44.893350 19449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } } }
I20260807 18:33:44.893622 19449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:44.894954 19462 catalog_manager.cc:1347] Generated new cluster ID: 06c244c8ad7b45f4b4d89e7e39b27198
I20260807 18:33:44.895058 19462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:44.897153 19340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:44.900018 19466 mvcc.cc:205] Tried to move back new op lower bound from 7315978751439011840 to 7315978751248445440. Current Snapshot: MvccSnapshot[applied={T|T < 7315978751439011840 or (T in {7315978751439011840})}]
I20260807 18:33:44.897213 19313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "03c9391be69644ea98cdbf8e1bfc937d"
I20260807 18:33:44.897832 19312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:44.905874 19312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [term 1 FOLLOWER]: Refusing update from remote peer 980f7bb423eb466497a8a7a9fe9a1aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:33:44.909688 19313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f7bb423eb466497a8a7a9fe9a1aad in term 1.
I20260807 18:33:44.911355 19428 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:33:44.928396 19462 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:44.930346 19462 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:44.952701 19470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980f7bb423eb466497a8a7a9fe9a1aad. Latest consensus state: current_term: 1 leader_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } } }
I20260807 18:33:44.953050 19470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:44.952782 19476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } } }
I20260807 18:33:44.953364 19476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:44.954260 19428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980f7bb423eb466497a8a7a9fe9a1aad" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 35871 } } peers { permanent_uuid: "03c9391be69644ea98cdbf8e1bfc937d" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 43787 } } peers { permanent_uuid: "8697b2c867d24292b6f42f4c2559e458" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 43805 } } }
I20260807 18:33:44.954656 19428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:44.971992 19462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 980f7bb423eb466497a8a7a9fe9a1aad: Generated new TSK 0
I20260807 18:33:44.973071 19462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:44.990600 19488 catalog_manager.cc:1259] Loaded cluster ID: 06c244c8ad7b45f4b4d89e7e39b27198
I20260807 18:33:44.990720 19488 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d: loading cluster ID for follower catalog manager: success
I20260807 18:33:44.994553 19488 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d: acquiring CA information for follower catalog manager: success
I20260807 18:33:44.997087 19488 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03c9391be69644ea98cdbf8e1bfc937d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 18:33:45.074914 19425 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.068690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.27.1.252:43805)
0807 18:33:45.068882 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.068889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.072968 (+ 4079us) server_negotiation.cc:407] Connection header received
0807 18:33:45.073174 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.073181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.073272 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.073385 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.074611 (+ 1226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51318: BlockingRecv error: recv got EOF from 127.0.0.1:51318 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 18:33:45.077143 19423 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.069219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.27.1.254:35871)
0807 18:33:45.069390 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.069396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.074161 (+ 4765us) server_negotiation.cc:407] Connection header received
0807 18:33:45.075650 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.075656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.075733 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.075803 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.076894 (+ 1091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37636: BlockingRecv error: recv got EOF from 127.0.0.1:37636 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260807 18:33:45.077250 19346 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.074686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.27.1.253:43787)
0807 18:33:45.074861 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.074867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.075658 (+ 791us) server_negotiation.cc:407] Connection header received
0807 18:33:45.075951 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.075957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.076054 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.076123 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.077013 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49074: BlockingRecv error: recv got EOF from 127.0.0.1:49074 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 18:33:45.092216 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.084652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.27.1.253:43787)
0807 18:33:45.084835 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.084841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.085119 (+ 278us) server_negotiation.cc:407] Connection header received
0807 18:33:45.085332 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.085338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.085540 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.085697 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.086149 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.087320 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.089342 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.090031 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.090163 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.090746 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.091524 (+ 778us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.091678 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.091801 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.091885 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260807 18:33:45.103742 19425 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.101336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.27.1.252:43805)
0807 18:33:45.101549 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.101555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.101642 (+ 87us) server_negotiation.cc:407] Connection header received
0807 18:33:45.102283 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.102288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.102431 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.102546 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.103515 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51320: BlockingRecv error: recv got EOF from 127.0.0.1:51320 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 18:33:45.148133 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.141425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.27.1.253:43787)
0807 18:33:45.141606 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.141612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.141635 (+ 23us) server_negotiation.cc:407] Connection header received
0807 18:33:45.141848 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.141854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.141970 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.142041 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.142774 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.143990 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.145956 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.146679 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.146805 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.147320 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.147477 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.147575 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.147744 (+ 169us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.147827 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 18:33:45.161293 19425 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.155571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.27.1.252:43805)
0807 18:33:45.155758 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.155764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.159740 (+ 3976us) server_negotiation.cc:407] Connection header received
0807 18:33:45.159933 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.159940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.160036 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.160146 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.161052 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51328: BlockingRecv error: recv got EOF from 127.0.0.1:51328 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:33:45.174723 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.167557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.27.1.253:43787)
0807 18:33:45.167750 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.167756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.168254 (+ 498us) server_negotiation.cc:407] Connection header received
0807 18:33:45.168519 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.168524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.168607 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.168666 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.169333 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.170549 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.172674 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.173330 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.173449 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.173912 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.174065 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.174157 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.174317 (+ 160us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.174401 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 18:33:45.189225 19425 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.185072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.27.1.252:43805)
0807 18:33:45.185255 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.185260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.187726 (+ 2466us) server_negotiation.cc:407] Connection header received
0807 18:33:45.187907 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.187913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.187999 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.188105 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.189001 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51338: BlockingRecv error: recv got EOF from 127.0.0.1:51338 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 18:33:45.203981 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.197052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.27.1.253:43787)
0807 18:33:45.197234 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.197240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.197774 (+ 534us) server_negotiation.cc:407] Connection header received
0807 18:33:45.197963 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.197969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.198057 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.198154 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.198653 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.199942 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.201805 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.202519 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.202639 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.203114 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.203270 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.203364 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.203564 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.203657 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 18:33:45.215390 19425 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.213361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.27.1.252:43805)
0807 18:33:45.213610 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.213616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.213637 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:33:45.213854 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.213859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.213983 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.214062 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.215143 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51350: BlockingRecv error: recv got EOF from 127.0.0.1:51350 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
I20260807 18:33:45.234447 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.227237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.27.1.253:43787)
0807 18:33:45.227446 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.227451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.227962 (+ 511us) server_negotiation.cc:407] Connection header received
0807 18:33:45.228140 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.228147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.228248 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.228349 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.228820 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.229954 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.231922 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.232785 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.232919 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.233488 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.233636 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.233734 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.233952 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.234037 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 18:33:45.253499 19425 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.249034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.27.1.252:43805)
0807 18:33:45.249229 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.249235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.251741 (+ 2506us) server_negotiation.cc:407] Connection header received
0807 18:33:45.251957 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.251963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.252067 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.252205 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.253265 (+ 1060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51354: BlockingRecv error: recv got EOF from 127.0.0.1:51354 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 18:33:45.280992 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.273465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.27.1.253:43787)
0807 18:33:45.273669 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.273678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.274235 (+ 557us) server_negotiation.cc:407] Connection header received
0807 18:33:45.274458 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.274464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.274570 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.274686 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.275342 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.276675 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.278698 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.279444 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.279591 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.280132 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.280324 (+ 192us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.280444 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.280617 (+ 173us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.280706 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260807 18:33:45.299360 19425 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.297203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.27.1.252:43805)
0807 18:33:45.297393 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.297400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.297443 (+ 43us) server_negotiation.cc:407] Connection header received
0807 18:33:45.297708 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.297713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.297815 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.297900 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.299052 (+ 1152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51370: BlockingRecv error: recv got EOF from 127.0.0.1:51370 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 18:33:45.343480 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.335961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.27.1.253:43787)
0807 18:33:45.336165 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.336172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.336804 (+ 632us) server_negotiation.cc:407] Connection header received
0807 18:33:45.337012 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.337020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.337135 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.337245 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.337825 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.339072 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.341066 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.341865 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.342003 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.342532 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.342678 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.342785 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.342972 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.343065 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 18:33:45.355892 19425 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.353123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.27.1.252:43805)
0807 18:33:45.353319 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.353326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.353877 (+ 551us) server_negotiation.cc:407] Connection header received
0807 18:33:45.354041 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.354047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.354157 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.354266 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.355620 (+ 1354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51374: BlockingRecv error: recv got EOF from 127.0.0.1:51374 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:33:45.441574 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.436537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.27.1.253:43787)
0807 18:33:45.436720 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.436725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.437132 (+ 407us) server_negotiation.cc:407] Connection header received
0807 18:33:45.437346 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.437353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.437456 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.437576 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.437954 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.438694 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.440076 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.440532 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.440641 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.440987 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.441095 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.441161 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.441292 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.441358 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 18:33:45.449303 19425 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.447424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.27.1.252:43805)
0807 18:33:45.447589 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.447594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.448035 (+ 441us) server_negotiation.cc:407] Connection header received
0807 18:33:45.448168 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.448172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.448254 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.448352 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.449096 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51378: BlockingRecv error: recv got EOF from 127.0.0.1:51378 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 18:33:45.586566 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.580780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49140 (local address 127.27.1.253:43787)
0807 18:33:45.580988 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.580995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.581399 (+ 404us) server_negotiation.cc:407] Connection header received
0807 18:33:45.581565 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.581571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.581657 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.581764 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.582263 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.583408 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.584752 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.585374 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.585495 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.585861 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.585991 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.586075 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.586201 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.586278 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 18:33:45.594151 19425 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:45.592244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.27.1.252:43805)
0807 18:33:45.592446 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.592452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.592747 (+ 295us) server_negotiation.cc:407] Connection header received
0807 18:33:45.592908 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.592911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.592982 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.593064 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:45.593905 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51386: BlockingRecv error: recv got EOF from 127.0.0.1:51386 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 18:33:45.834146 19460 catalog_manager.cc:1259] Loaded cluster ID: 06c244c8ad7b45f4b4d89e7e39b27198
I20260807 18:33:45.834340 19460 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458: loading cluster ID for follower catalog manager: success
I20260807 18:33:45.837039 19460 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458: acquiring CA information for follower catalog manager: success
I20260807 18:33:45.839161 19460 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8697b2c867d24292b6f42f4c2559e458: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 18:33:45.860013 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.854134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.27.1.253:43787)
0807 18:33:45.854399 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.854425 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.854789 (+ 364us) server_negotiation.cc:407] Connection header received
0807 18:33:45.854956 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.854961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.855053 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.855197 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.855654 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.856801 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.858121 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.858767 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.858896 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.859300 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.859404 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.859510 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.859671 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.859751 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 18:33:45.878742 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.871981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.27.1.252:43805)
0807 18:33:45.872220 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.872226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.872581 (+ 355us) server_negotiation.cc:407] Connection header received
0807 18:33:45.872737 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.872743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.872938 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.873078 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.873460 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.874608 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.876011 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.876740 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.876879 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.877364 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.878076 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.878234 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.878284 (+ 50us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.878385 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 18:33:45.905114 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.885776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.27.1.254:35871)
0807 18:33:45.896052 (+ 10276us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.896058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.896935 (+ 877us) server_negotiation.cc:407] Connection header received
0807 18:33:45.897069 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.897075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.897280 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.897467 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.898320 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.899483 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.901821 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.902494 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.902614 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.903405 (+ 791us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.903898 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.904011 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.904700 (+ 689us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.904794 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10026,"thread_start_us":10118,"threads_started":1}
I20260807 18:33:45.906474 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.886026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.27.1.252:43805)
0807 18:33:45.887626 (+ 1600us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.887631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.897054 (+ 9423us) server_negotiation.cc:407] Connection header received
0807 18:33:45.897329 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.897334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.897421 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.897491 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.898600 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.900182 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.903222 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.904088 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.904262 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.905171 (+ 909us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.905353 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.905437 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.905950 (+ 513us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.906081 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1470}
I20260807 18:33:45.912730 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:45.897146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.27.1.253:43787)
0807 18:33:45.897334 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:45.897340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:45.904615 (+ 7275us) server_negotiation.cc:407] Connection header received
0807 18:33:45.905632 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:45.905638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:45.905731 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:45.905799 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:45.906376 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.907485 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.910347 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:45.911195 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:45.911329 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:45.911919 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:45.912039 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:45.912141 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:45.912332 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 18:33:45.912421 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 18:33:45.912829 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 19193
I20260807 18:33:45.913960 19259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:46.047111 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 19193
I20260807 18:33:46.073446 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 19271
I20260807 18:33:46.074689 19335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:46.184700 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 19271
I20260807 18:33:46.216451 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 19349
I20260807 18:33:46.217650 19414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:46.336383 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 19349
2026-08-07T18:33:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4743 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 18:33:46 dist-test-slave-1xrh krb5kdc[19627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 18:33:46 dist-test-slave-1xrh krb5kdc[19627](info): set up 2 sockets
Aug 07 18:33:46 dist-test-slave-1xrh krb5kdc[19627](info): commencing operation
krb5kdc: starting...
W20260807 18:33:47.513399 27655 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/test-user.keytab.
Aug 07 18:33:47 dist-test-slave-1xrh krb5kdc[19627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T18:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:33:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
Entry for principal kudu/127.27.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab.
I20260807 18:33:47.834712 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:45751
--webserver_interface=127.27.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:45751,127.27.1.253:42877,127.27.1.252:36793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 18:33:48.176074 19641 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 18:33:48.176452 19641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:48.176581 19641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:48.176715 19641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:48.183991 19641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:48.184074 19641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:48.184208 19641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:48.184285 19641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:48.184371 19641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:48.194295 19641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:45751,127.27.1.253:42877,127.27.1.252:36793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.254.keytab
--principal=kudu/127.27.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.254:45751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:48.195724 19641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:48.197083 19641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:48.208834 19649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:48.208952 19646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:48.208801 19647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:48.209539 19641 server_base.cc:1034] running on GCE node
I20260807 18:33:48.210319 19641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:48.211804 19641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:48.213055 19641 hybrid_clock.cc:648] HybridClock initialized: now 1786127628213003 us; error 41 us; skew 500 ppm
I20260807 18:33:48.213382 19641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:48 dist-test-slave-1xrh krb5kdc[19627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:48.216552 19641 init.cc:377] Logged in from keytab as kudu/127.27.1.254@KRBTEST.COM (short username kudu)
I20260807 18:33:48.217948 19641 webserver.cc:460] Webserver started at http://127.27.1.254:42513/ using document root <none> and password file <none>
I20260807 18:33:48.218642 19641 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:48.218775 19641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:48.219105 19641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:48.222092 19641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/data/instance:
uuid: "bd905673f48b4d17a56be10d35d83105"
format_stamp: "Formatted at 2026-08-07 18:33:48 on dist-test-slave-1xrh"
server_key: "9f45a5504127fdaee11a58a4f229aa19"
server_key_iv: "425e476b117ef32097641b9eadeb3925"
server_key_version: "encryptionkey@0"
I20260807 18:33:48.222903 19641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/wal/instance:
uuid: "bd905673f48b4d17a56be10d35d83105"
format_stamp: "Formatted at 2026-08-07 18:33:48 on dist-test-slave-1xrh"
server_key: "9f45a5504127fdaee11a58a4f229aa19"
server_key_iv: "425e476b117ef32097641b9eadeb3925"
server_key_version: "encryptionkey@0"
I20260807 18:33:48.228008 19641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260807 18:33:48.231460 19656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:48.232697 19641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 18:33:48.232893 19641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/wal
uuid: "bd905673f48b4d17a56be10d35d83105"
format_stamp: "Formatted at 2026-08-07 18:33:48 on dist-test-slave-1xrh"
server_key: "9f45a5504127fdaee11a58a4f229aa19"
server_key_iv: "425e476b117ef32097641b9eadeb3925"
server_key_version: "encryptionkey@0"
I20260807 18:33:48.233109 19641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:48.249001 19641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:48.252921 19641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:48.253202 19641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:48.272900 19641 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.254:45751
I20260807 18:33:48.272924 19711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.254:45751 every 8 connection(s)
I20260807 18:33:48.274228 19641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/data/info.pb
I20260807 18:33:48.276444 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 19641
I20260807 18:33:48.276659 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-0/wal/instance
I20260807 18:33:48.277246 27655 external_mini_cluster.cc:1428] Setting key b56f8f7a6b0dd784cb30728ed8038033
I20260807 18:33:48.278561 19712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:48.288611 19712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:33:48 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.1.253@KRBTEST.COM; defaulting to no policy
I20260807 18:33:48.311363 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.292445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:45751 (local address 127.0.0.1:57516)
0807 18:33:48.293513 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.293556 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:48.295387 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:48.296201 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:48.296208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:48.296893 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:48.297225 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:48.297242 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.298827 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.298831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:48.299928 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:48.299941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.300872 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.300879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:48.301067 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:48.301775 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:48.301797 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:48.304038 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:48.307198 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:48.307208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:48.307219 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:48.307608 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:48.308032 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:48.308036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:48.308040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:48.308201 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:48.308924 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:48.308931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:48.309166 (+ 235us) client_negotiation.cc:769] Sending connection context
0807 18:33:48.309406 (+ 240us) client_negotiation.cc:240] Negotiation successful
0807 18:33:48.310009 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":149,"thread_start_us":552,"threads_started":1}
I20260807 18:33:48.311383 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.292445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.27.1.254:45751)
0807 18:33:48.293403 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:48.293426 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:48.293683 (+ 257us) server_negotiation.cc:407] Connection header received
0807 18:33:48.295746 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:48.295752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:48.295976 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:48.296176 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:48.297446 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.298647 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.300084 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.300623 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.300822 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:48.304257 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:48.304265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:48.304276 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:48.304303 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:48.307040 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:48.307752 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:48.307759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:48.307761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:48.307900 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:48.308319 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:48.308330 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:48.308333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:48.308790 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:48.309023 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:48.309560 (+ 537us) server_negotiation.cc:299] Negotiation successful
0807 18:33:48.309862 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":486,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.253@KRBTEST.COM" created.
I20260807 18:33:48.314452 19712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } has no permanent_uuid. Determining permanent_uuid...
I20260807 18:33:48.315659 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.315270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51560)
0807 18:33:48.315440 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.315544 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 18:33:48.315936 19660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
I20260807 18:33:48.317502 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.317144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51574)
0807 18:33:48.317320 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.317384 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 18:33:48.317713 19712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:42877: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab.
Entry for principal kudu/127.27.1.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab.
I20260807 18:33:48.350003 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.253:42877
--webserver_interface=127.27.1.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab
--principal=kudu/127.27.1.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:45751,127.27.1.253:42877,127.27.1.252:36793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:33:48.367049 19712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } attempt: 1
I20260807 18:33:48.368432 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.367830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51584)
0807 18:33:48.368097 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.368216 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260807 18:33:48.369611 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.369156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51598)
0807 18:33:48.369344 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.369434 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 18:33:48.369802 19712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:42877: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
I20260807 18:33:48.438054 19712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } attempt: 2
I20260807 18:33:48.439389 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.438747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51610)
0807 18:33:48.439053 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.439212 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260807 18:33:48.440470 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.440058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51612)
0807 18:33:48.440210 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.440312 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 18:33:48.440729 19712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:42877: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
I20260807 18:33:48.532037 19712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } attempt: 3
I20260807 18:33:48.533336 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.532792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51618)
0807 18:33:48.533039 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.533152 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 18:33:48.534448 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.533977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51628)
0807 18:33:48.534142 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.534239 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 18:33:48.534668 19712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:42877: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
I20260807 18:33:48.677953 19712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } attempt: 4
I20260807 18:33:48.679455 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.678691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51644)
0807 18:33:48.679111 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.679251 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260807 18:33:48.680605 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.680177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51658)
0807 18:33:48.680344 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.680445 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 18:33:48.680799 19712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.253:42877: Network error: Client connection negotiation failed: client connection to 127.27.1.253:42877: connect: Connection refused (error 111)
W20260807 18:33:48.686846 19720 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 18:33:48.687232 19720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:48.687353 19720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:48.687513 19720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:48.695008 19720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:48.695130 19720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:48.695271 19720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:48.695356 19720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:48.695457 19720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:48.705549 19720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:45751,127.27.1.253:42877,127.27.1.252:36793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.253.keytab
--principal=kudu/127.27.1.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.253:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.1.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:48.706845 19720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:48.708232 19720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:48.720453 19726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:48.720489 19728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:48.720453 19725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:48.721120 19720 server_base.cc:1034] running on GCE node
I20260807 18:33:48.721863 19720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:48.723294 19720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:48.724543 19720 hybrid_clock.cc:648] HybridClock initialized: now 1786127628724486 us; error 48 us; skew 500 ppm
I20260807 18:33:48.724875 19720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:48 dist-test-slave-1xrh krb5kdc[19627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:48.727973 19720 init.cc:377] Logged in from keytab as kudu/127.27.1.253@KRBTEST.COM (short username kudu)
I20260807 18:33:48.729442 19720 webserver.cc:460] Webserver started at http://127.27.1.253:44829/ using document root <none> and password file <none>
I20260807 18:33:48.730176 19720 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:48.730314 19720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:48.730721 19720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:48.733680 19720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/data/instance:
uuid: "f1b16cc8a2a94429acee5f96c0ba4bca"
format_stamp: "Formatted at 2026-08-07 18:33:48 on dist-test-slave-1xrh"
server_key: "0ce130078c9b7a54ade44f7b2ad81e8c"
server_key_iv: "e0dbb90f253319bfe1a1a7e9e562a49f"
server_key_version: "encryptionkey@0"
I20260807 18:33:48.734464 19720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/wal/instance:
uuid: "f1b16cc8a2a94429acee5f96c0ba4bca"
format_stamp: "Formatted at 2026-08-07 18:33:48 on dist-test-slave-1xrh"
server_key: "0ce130078c9b7a54ade44f7b2ad81e8c"
server_key_iv: "e0dbb90f253319bfe1a1a7e9e562a49f"
server_key_version: "encryptionkey@0"
I20260807 18:33:48.739751 19720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 18:33:48.743113 19736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:48.744526 19720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 18:33:48.744696 19720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/wal
uuid: "f1b16cc8a2a94429acee5f96c0ba4bca"
format_stamp: "Formatted at 2026-08-07 18:33:48 on dist-test-slave-1xrh"
server_key: "0ce130078c9b7a54ade44f7b2ad81e8c"
server_key_iv: "e0dbb90f253319bfe1a1a7e9e562a49f"
server_key_version: "encryptionkey@0"
I20260807 18:33:48.744912 19720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:48.763921 19720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:48.768111 19720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:48.768391 19720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:48.785802 19720 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.253:42877
I20260807 18:33:48.785817 19789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.253:42877 every 8 connection(s)
I20260807 18:33:48.787197 19720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/data/info.pb
I20260807 18:33:48.790861 19790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:48.791733 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 19720
I20260807 18:33:48.791932 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-1/wal/instance
I20260807 18:33:48.792385 27655 external_mini_cluster.cc:1428] Setting key 26cb1a2da6b1507e87ce655100f234a6
I20260807 18:33:48.797269 19790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:33:48 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:33:48.818186 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.799762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.27.1.254:45751)
0807 18:33:48.799991 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:48.799997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:48.801239 (+ 1242us) server_negotiation.cc:407] Connection header received
0807 18:33:48.804145 (+ 2906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:48.804150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:48.804221 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:48.804301 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:48.806333 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.807007 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.808525 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.808954 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.809095 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:48.812495 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:48.812500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:48.812504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:48.812535 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:48.815200 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:48.815933 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:48.815938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:48.815941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:48.816057 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:48.816476 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:48.816486 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:48.816488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:48.816762 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:48.816929 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:48.817625 (+ 696us) server_negotiation.cc:299] Negotiation successful
0807 18:33:48.817848 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 18:33:48.819085 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.800228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:45751 (local address 127.0.0.1:57532)
0807 18:33:48.801040 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.801100 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:48.803835 (+ 2735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:48.804572 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:48.804585 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:48.805365 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:48.806154 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:48.806181 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.807206 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.807210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:48.808353 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:48.808365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.809187 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.809194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:48.809437 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:48.810115 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:48.810139 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:48.812271 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:48.815328 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:48.815337 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:48.815352 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:48.815787 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:48.816170 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:48.816175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:48.816178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:48.816350 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:48.816900 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:48.816908 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:48.817253 (+ 345us) client_negotiation.cc:769] Sending connection context
0807 18:33:48.817518 (+ 265us) client_negotiation.cc:240] Negotiation successful
0807 18:33:48.818001 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":332,"threads_started":1}
I20260807 18:33:48.820868 19790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.1.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.1.252@KRBTEST.COM" created.
Aug 07 18:33:48 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:33:48.836261 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.821720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51672)
0807 18:33:48.821918 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.821936 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:48.822021 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:48.823031 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:48.823035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:48.823500 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:48.823624 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:48.823632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.825141 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.825146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:48.826037 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:48.826049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.826937 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.826943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:48.827110 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:48.827785 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:48.827803 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:48.829705 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:48.833912 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:48.833916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:48.833918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:48.834277 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:48.834693 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:48.834696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:48.834698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:48.834828 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:48.835592 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:48.835595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:48.835708 (+ 113us) client_negotiation.cc:769] Sending connection context
0807 18:33:48.835818 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 18:33:48.836015 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 18:33:48.836758 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.821948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.27.1.253:42877)
0807 18:33:48.822656 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:48.822662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:48.822685 (+ 23us) server_negotiation.cc:407] Connection header received
0807 18:33:48.822725 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:48.822730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:48.822885 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:48.823044 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:48.823813 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.824991 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.826181 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.826755 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.826890 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:48.829931 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:48.829947 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:48.829967 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:48.830023 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:48.833780 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:48.834410 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:48.834421 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:48.834427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:48.834577 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:48.834940 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:48.834947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:48.834953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:48.835461 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:48.835679 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:48.836015 (+ 336us) server_negotiation.cc:299] Negotiation successful
0807 18:33:48.836291 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":417,"threads_started":1}
I20260807 18:33:48.839051 19790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } has no permanent_uuid. Determining permanent_uuid...
I20260807 18:33:48.840325 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.839738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44512)
0807 18:33:48.839955 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.840127 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 18:33:48.840593 19740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
I20260807 18:33:48.842294 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.841852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44514)
0807 18:33:48.842047 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.842160 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 18:33:48.842515 19790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:36793: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.1.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab.
Entry for principal kudu/127.27.1.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab.
I20260807 18:33:48.867457 27655 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
/tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.252:36793
--webserver_interface=127.27.1.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab
--principal=kudu/127.27.1.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.1.212:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.1.254:45751,127.27.1.253:42877,127.27.1.252:36793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 18:33:48.891832 19790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } attempt: 1
I20260807 18:33:48.893126 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.892575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44520)
0807 18:33:48.892786 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.892901 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 18:33:48.894223 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.893877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44534)
0807 18:33:48.894039 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.894124 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 18:33:48.894434 19790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:36793: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
I20260807 18:33:48.962648 19790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } attempt: 2
I20260807 18:33:48.963908 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.963446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44546)
0807 18:33:48.963643 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.963735 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 18:33:48.964949 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.964612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44556)
0807 18:33:48.964764 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.964839 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 18:33:48.965161 19790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:36793: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
I20260807 18:33:48.980059 19712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } attempt: 5
Aug 07 18:33:48 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:33:48.994845 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.980822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:51678)
0807 18:33:48.981053 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.981074 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:48.981210 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:48.981597 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:48.981602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:48.982130 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:48.982367 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:48.982380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.983412 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.983417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:48.984827 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:48.984843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.985617 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.985625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:48.985921 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:48.986635 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:48.986665 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:48.989254 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:48.992352 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:48.992358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:48.992363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:48.992744 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:48.993136 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:48.993141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:48.993145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:48.993288 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:48.993819 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:48.993824 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:48.993995 (+ 171us) client_negotiation.cc:769] Sending connection context
0807 18:33:48.994170 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 18:33:48.994460 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 18:33:48.994866 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.980938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.27.1.253:42877)
0807 18:33:48.981124 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:48.981128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:48.981149 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:33:48.981343 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:48.981347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:48.981429 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:48.981528 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:48.982525 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.983277 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.984980 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:48.985429 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:48.985542 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:48.989464 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:48.989489 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:48.989493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:48.989527 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:48.992207 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:48.992869 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:48.992877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:48.992880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:48.993014 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:48.993401 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:48.993407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:48.993410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:48.993695 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:48.993852 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:48.994274 (+ 422us) server_negotiation.cc:299] Negotiation successful
0807 18:33:48.994514 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:33:48.996308 19712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } has no permanent_uuid. Determining permanent_uuid...
I20260807 18:33:48.997342 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.996954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44568)
0807 18:33:48.997119 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.997199 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 18:33:48.998289 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:48.997986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44572)
0807 18:33:48.998118 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:48.998199 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 18:33:48.998555 19712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:36793: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
I20260807 18:33:49.049777 19712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } attempt: 1
I20260807 18:33:49.051023 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.050555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44580)
0807 18:33:49.050751 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.050844 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 18:33:49.052248 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.051935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44594)
0807 18:33:49.052082 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.052150 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260807 18:33:49.052553 19712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:36793: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
I20260807 18:33:49.056424 19790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } attempt: 3
I20260807 18:33:49.057587 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.057012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44602)
0807 18:33:49.057284 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.057391 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260807 18:33:49.059224 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.058534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44606)
0807 18:33:49.058948 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.059059 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
W20260807 18:33:49.059571 19790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:36793: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
I20260807 18:33:49.120781 19712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } attempt: 2
I20260807 18:33:49.122041 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.121550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44618)
0807 18:33:49.121779 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.121868 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 18:33:49.123226 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.122833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44628)
0807 18:33:49.122986 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.123054 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 18:33:49.123456 19712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:36793: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
I20260807 18:33:49.202828 19790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } attempt: 4
I20260807 18:33:49.204144 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.203560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44640)
0807 18:33:49.203841 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.203953 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 18:33:49.205258 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.204879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44652)
0807 18:33:49.205039 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.205127 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 18:33:49.205495 19790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:36793: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
W20260807 18:33:49.214197 19798 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 18:33:49.214540 19798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:33:49.214653 19798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:33:49.214782 19798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:33:49.221920 19798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:33:49.221999 19798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 18:33:49.222144 19798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:33:49.222224 19798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:33:49.222332 19798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:33:49.229684 19712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } attempt: 3
I20260807 18:33:49.230800 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.230368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44660)
0807 18:33:49.230556 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.230642 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 18:33:49.231905 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.231570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:44676)
0807 18:33:49.231728 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.231797 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 18:33:49.232167 19712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.1.252:36793: Network error: Client connection negotiation failed: client connection to 127.27.1.252:36793: connect: Connection refused (error 111)
I20260807 18:33:49.233457 19798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.1.212:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.1.254:45751,127.27.1.253:42877,127.27.1.252:36793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/krb5kdc/kudu_127.27.1.252.keytab
--principal=kudu/127.27.1.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.1.252:36793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.1.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:33:49.234763 19798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:33:49.236203 19798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:33:49.247962 19804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:49.248133 19805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:33:49.248128 19808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:33:49.249205 19798 server_base.cc:1034] running on GCE node
I20260807 18:33:49.250005 19798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:33:49.251590 19798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:33:49.252864 19798 hybrid_clock.cc:648] HybridClock initialized: now 1786127629252782 us; error 35 us; skew 500 ppm
I20260807 18:33:49.253196 19798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 18:33:49 dist-test-slave-1xrh krb5kdc[19627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 18:33:49.256454 19798 init.cc:377] Logged in from keytab as kudu/127.27.1.252@KRBTEST.COM (short username kudu)
I20260807 18:33:49.257851 19798 webserver.cc:460] Webserver started at http://127.27.1.252:35569/ using document root <none> and password file <none>
I20260807 18:33:49.258561 19798 fs_manager.cc:359] Metadata directory not provided
I20260807 18:33:49.258695 19798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:33:49.259030 19798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:33:49.262109 19798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/data/instance:
uuid: "b492e06edd7941d682d38ef97a8ccd49"
format_stamp: "Formatted at 2026-08-07 18:33:49 on dist-test-slave-1xrh"
server_key: "504793620a7d0272eca351b8422caf46"
server_key_iv: "5b047af81af45777ee1b7e5115e9cb42"
server_key_version: "encryptionkey@0"
I20260807 18:33:49.262912 19798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/wal/instance:
uuid: "b492e06edd7941d682d38ef97a8ccd49"
format_stamp: "Formatted at 2026-08-07 18:33:49 on dist-test-slave-1xrh"
server_key: "504793620a7d0272eca351b8422caf46"
server_key_iv: "5b047af81af45777ee1b7e5115e9cb42"
server_key_version: "encryptionkey@0"
I20260807 18:33:49.268007 19798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260807 18:33:49.271353 19815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:33:49.272647 19798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 18:33:49.272840 19798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/data,/tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/wal
uuid: "b492e06edd7941d682d38ef97a8ccd49"
format_stamp: "Formatted at 2026-08-07 18:33:49 on dist-test-slave-1xrh"
server_key: "504793620a7d0272eca351b8422caf46"
server_key_iv: "5b047af81af45777ee1b7e5115e9cb42"
server_key_version: "encryptionkey@0"
I20260807 18:33:49.273074 19798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:33:49.298983 19798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:33:49.302997 19798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:33:49.303306 19798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:33:49.328815 19798 rpc_server.cc:307] RPC server started. Bound to: 127.27.1.252:36793
I20260807 18:33:49.328823 19869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.1.252:36793 every 8 connection(s)
I20260807 18:33:49.330209 19798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/data/info.pb
I20260807 18:33:49.330991 27655 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu as pid 19798
I20260807 18:33:49.331231 27655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrFJVJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786127255008662-27655-0/minicluster-data/master-2/wal/instance
I20260807 18:33:49.331820 27655 external_mini_cluster.cc:1428] Setting key 7a6db94820572858c6897b926806856c
I20260807 18:33:49.334869 19870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:33:49.341939 19870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:33:49 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
I20260807 18:33:49.354445 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.333693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.27.1.252:36793)
0807 18:33:49.334415 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.334427 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.334468 (+ 41us) server_negotiation.cc:407] Connection header received
0807 18:33:49.336270 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.336292 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.336659 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.337019 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.338553 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.339973 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.341283 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.341942 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.342146 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.346322 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.346335 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.346357 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.346421 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.350265 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.351110 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.351116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.351121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.351330 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.351793 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.351796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.351798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.352364 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.352638 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.353016 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.353381 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":304,"threads_started":1}
Aug 07 18:33:49 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
I20260807 18:33:49.363615 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.344391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.27.1.254:45751)
0807 18:33:49.347473 (+ 3082us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.347481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.347503 (+ 22us) server_negotiation.cc:407] Connection header received
0807 18:33:49.347566 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.347571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.347654 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.347777 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.349258 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.350425 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.352264 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.352938 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.353071 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.356944 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.356949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.356952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.356984 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.360784 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.361497 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.361504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.361507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.361641 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.362047 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.362053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.362056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.362315 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.362485 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.362957 (+ 472us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.363184 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2939,"thread_start_us":2835,"threads_started":1}
I20260807 18:33:49.363615 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.344376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.254:45751 (local address 127.0.0.1:57542)
0807 18:33:49.346761 (+ 2385us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.346809 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:49.346965 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:49.347831 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:49.347836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:49.348754 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:49.349095 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:49.349105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.350581 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.350586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:49.352124 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:49.352137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.353212 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.353221 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:49.353427 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.354172 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:49.354195 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:49.356738 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:49.360928 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:49.360943 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:49.360949 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:49.361373 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:49.361752 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:49.361757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:49.361759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:49.361933 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:49.362461 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:49.362468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:49.362651 (+ 183us) client_negotiation.cc:769] Sending connection context
0807 18:33:49.362830 (+ 179us) client_negotiation.cc:240] Negotiation successful
0807 18:33:49.363168 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2125,"thread_start_us":1944,"threads_started":1}
I20260807 18:33:49.365454 19870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:33:49 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.254@KRBTEST.COM
Aug 07 18:33:49 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
I20260807 18:33:49.373390 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.357833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.27.1.254:45751)
0807 18:33:49.358433 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.358437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.358457 (+ 20us) server_negotiation.cc:407] Connection header received
0807 18:33:49.358499 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.358502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.358557 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.358650 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.360336 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.361333 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.362592 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.363052 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.363226 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.367814 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.367819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.367822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.367851 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.370400 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.371060 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.371076 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.371084 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.371253 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.371638 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.371644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.371647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.371919 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.372200 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.372953 (+ 753us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.373120 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":262,"threads_started":1}
I20260807 18:33:49.379004 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.366246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.253:42877 (local address 127.0.0.1:56812)
0807 18:33:49.366599 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.366618 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:49.366776 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:49.367205 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:49.367208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:49.367525 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:49.367655 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:49.367663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.368619 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.368625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:49.369498 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:49.369510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.370215 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.370221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:49.370392 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.370920 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:49.370936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:49.372802 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:49.376523 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:49.376528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:49.376532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:49.376883 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:49.377305 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:49.377308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:49.377310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:49.377514 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:49.378075 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:49.378080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:49.378242 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 18:33:49.378383 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 18:33:49.378658 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260807 18:33:49.379092 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.366235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.27.1.253:42877)
0807 18:33:49.366464 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.366471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.366684 (+ 213us) server_negotiation.cc:407] Connection header received
0807 18:33:49.366909 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.366916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.367002 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.367110 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.367804 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.368477 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.369644 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.370064 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.370203 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.372996 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.373001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.373005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.373036 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.376273 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.377006 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.377013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.377018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.377188 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.377640 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.377646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.377649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.377932 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.378112 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.378507 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.378719 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 18:33:49.380016 19870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 18:33:49 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.1.253@KRBTEST.COM
Aug 07 18:33:49 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.252@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
I20260807 18:33:49.391494 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.375339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.27.1.253:42877)
0807 18:33:49.375976 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.375980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.375999 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:33:49.376036 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.376040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.376092 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.376175 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.377713 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.378499 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.379943 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.380425 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.380559 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.385593 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.385600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.385605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.385643 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.388590 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.389270 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.389276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.389280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.389432 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.389853 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.389858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.389862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.390214 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.390490 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.390806 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.391048 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":296,"threads_started":1}
I20260807 18:33:49.394578 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.380651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:40440)
0807 18:33:49.380882 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.380904 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:49.380978 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:49.381329 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:49.381332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:49.381749 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:49.381875 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:49.381883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.383374 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.383377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:49.384574 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:49.384587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.385576 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.385582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:49.385781 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.386303 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:49.386319 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:49.388663 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:49.392027 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:49.392032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:49.392035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:49.392390 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:49.392819 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:49.392824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:49.392827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:49.392998 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:49.393603 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:49.393606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:49.393720 (+ 114us) client_negotiation.cc:769] Sending connection context
0807 18:33:49.393906 (+ 186us) client_negotiation.cc:240] Negotiation successful
0807 18:33:49.394216 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260807 18:33:49.394970 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.380766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.27.1.252:36793)
0807 18:33:49.380964 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.380972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.380993 (+ 21us) server_negotiation.cc:407] Connection header received
0807 18:33:49.381109 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.381113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.381165 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.381286 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.382021 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.383245 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.384712 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.385400 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.385529 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.388828 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.388833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.388836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.388863 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.391897 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.392543 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.392548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.392550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.392701 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.393110 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.393116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.393118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.393486 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.393628 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.394018 (+ 390us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.394437 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":59}
I20260807 18:33:49.396642 27655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 18:33:49.400673 19870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49: Bootstrap starting.
I20260807 18:33:49.399710 19687 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bd905673f48b4d17a56be10d35d83105"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43338
I20260807 18:33:49.403925 19870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:49.404943 19870 log.cc:824] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:49.407646 19870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49: No bootstrap required, opened a new log
I20260807 18:33:49.409401 19712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } attempt: 4
I20260807 18:33:49.416514 19870 raft_consensus.cc:348] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } }
I20260807 18:33:49.416920 19870 raft_consensus.cc:729] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b492e06edd7941d682d38ef97a8ccd49, State: Initialized, Role: FOLLOWER
I20260807 18:33:49.417362 19870 consensus_queue.cc:260] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } }
Aug 07 18:33:49 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.254@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
I20260807 18:33:49.418211 19881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } } }
I20260807 18:33:49.418607 19881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:49.419593 19870 sys_catalog.cc:564] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:49.426608 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.410063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:40450)
0807 18:33:49.410235 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.410253 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:49.410363 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:49.410710 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:49.410714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:49.411284 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:49.411480 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:49.411495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.412965 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.412978 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:49.414426 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:49.414441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.415511 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.415517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:49.415738 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.416394 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:49.416411 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:49.419090 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:49.423614 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:49.423622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:49.423626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:49.424149 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:49.424555 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:49.424561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:49.424566 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:49.424729 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:49.425558 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:49.425565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:49.425753 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 18:33:49.425898 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 18:33:49.426191 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 18:33:49.426615 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.410149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.27.1.252:36793)
0807 18:33:49.410341 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.410346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.410365 (+ 19us) server_negotiation.cc:407] Connection header received
0807 18:33:49.410487 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.410492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.410567 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.410677 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.411667 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.412824 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.414587 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.415343 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.415503 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.419306 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.419313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.419318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.419352 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.423418 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.424273 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.424279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.424282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.424425 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.424843 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.424848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.424851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.425300 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.425485 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.426027 (+ 542us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.426228 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260807 18:33:49.430092 19894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 18:33:49.430213 19894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 18:33:49.432713 19712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105: Bootstrap starting.
I20260807 18:33:49.427822 19798 master_runner.cc:186] Error getting master registration for 127.27.1.254:45751: OK, instance_id { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" instance_seqno: 1786127628267460 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 18:33:49.436142 19798 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 18:33:49.436859 19712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:49.438221 19712 log.cc:824] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:49.441620 19712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105: No bootstrap required, opened a new log
I20260807 18:33:49.449574 19712 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } }
I20260807 18:33:49.450024 19712 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd905673f48b4d17a56be10d35d83105, State: Initialized, Role: FOLLOWER
I20260807 18:33:49.450585 19712 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } }
I20260807 18:33:49.451844 19897 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } } }
I20260807 18:33:49.452266 19897 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:49.453075 19712 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:49.453191 19687 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 18:33:49.453415 19687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:49.456311 19687 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } }
I20260807 18:33:49.457593 19687 leader_election.cc:288] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [CANDIDATE]: Term 1 election: Requested vote from peers f1b16cc8a2a94429acee5f96c0ba4bca (127.27.1.253:42877), b492e06edd7941d682d38ef97a8ccd49 (127.27.1.252:36793)
I20260807 18:33:49.459280 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd905673f48b4d17a56be10d35d83105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b492e06edd7941d682d38ef97a8ccd49"
I20260807 18:33:49.459915 19844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:49.465668 19844 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd905673f48b4d17a56be10d35d83105 in term 1.
I20260807 18:33:49.466724 19660 leader_election.cc:302] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b492e06edd7941d682d38ef97a8ccd49, bd905673f48b4d17a56be10d35d83105; no voters:
I20260807 18:33:49.467347 19897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:33:49.471071 19897 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [term 1 LEADER]: Becoming Leader. State: Replica: bd905673f48b4d17a56be10d35d83105, State: Running, Role: LEADER
I20260807 18:33:49.471796 19897 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } }
I20260807 18:33:49.504662 19790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } attempt: 5
I20260807 18:33:49.504662 19907 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd905673f48b4d17a56be10d35d83105. Latest consensus state: current_term: 1 leader_uuid: "bd905673f48b4d17a56be10d35d83105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } } }
I20260807 18:33:49.505108 19907 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [sys.catalog]: This master's current role is: LEADER
W20260807 18:33:49.505944 19662 tablet.cc:2369] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:33:49.513118 19914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:33:49.523729 19914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 07 18:33:49 dist-test-slave-1xrh krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786127628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.1.253@KRBTEST.COM for kudu/127.27.1.252@KRBTEST.COM
W20260807 18:33:49.555383 19821 tablet.cc:2369] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:33:49.572474 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.489803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.27.1.253:42877)
0807 18:33:49.490039 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.490046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.502108 (+ 12062us) server_negotiation.cc:407] Connection header received
0807 18:33:49.502281 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.502287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.502374 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.502476 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.525048 (+ 22572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.526087 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.535029 (+ 8942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.535700 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.535801 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.554117 (+ 18316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.554126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.554131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.554172 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.557694 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.563233 (+ 5539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.563239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.563243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.563388 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.569183 (+ 5795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.569189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.569192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.569559 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.569801 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.571856 (+ 2055us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.572114 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 18:33:49.580467 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.505261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.1.252:36793 (local address 127.0.0.1:40470)
0807 18:33:49.505550 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0807 18:33:49.505573 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 18:33:49.505691 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:33:49.521303 (+ 15612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 18:33:49.521309 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 18:33:49.521842 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0807 18:33:49.522076 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:49.522092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.530560 (+ 8468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.530567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:49.532047 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 18:33:49.532062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.536750 (+ 4688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.536757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 18:33:49.536958 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.537568 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 18:33:49.537595 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 18:33:49.540522 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 18:33:49.575279 (+ 34757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:49.575285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:49.575289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:49.575821 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:49.578066 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 18:33:49.578072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 18:33:49.578076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 18:33:49.578275 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 18:33:49.579043 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 18:33:49.579052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 18:33:49.579303 (+ 251us) client_negotiation.cc:769] Sending connection context
0807 18:33:49.579504 (+ 201us) client_negotiation.cc:240] Negotiation successful
0807 18:33:49.579895 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260807 18:33:49.581485 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.502922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.27.1.252:36793)
0807 18:33:49.503109 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.503115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.516186 (+ 13071us) server_negotiation.cc:407] Connection header received
0807 18:33:49.516356 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.516365 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.516483 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.516604 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.522829 (+ 6225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.523988 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.547150 (+ 23162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.547856 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.547962 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.563960 (+ 15998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.563967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.563981 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.564019 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.567750 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.572838 (+ 5088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.572844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.572847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.573026 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.579178 (+ 6152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.579183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.579187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.579529 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.579753 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.580861 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.581139 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 18:33:49.582311 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.482809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.27.1.254:45751)
0807 18:33:49.483037 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.483044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.501326 (+ 18282us) server_negotiation.cc:407] Connection header received
0807 18:33:49.501589 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.501594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.501693 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.501820 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.523548 (+ 21728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.524682 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.533125 (+ 8443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.533838 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.533946 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.558651 (+ 24705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.558659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.558664 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.558710 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.562635 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.568838 (+ 6203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.568844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.568848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.569011 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.570580 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.570586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.570589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.570989 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.571277 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.577382 (+ 6105us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.577804 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 18:33:49.583241 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.506121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.27.1.252:36793)
0807 18:33:49.520918 (+ 14797us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.520924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.520948 (+ 24us) server_negotiation.cc:407] Connection header received
0807 18:33:49.521015 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.521021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.521100 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.522244 (+ 1144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.522518 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.529036 (+ 6518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.532242 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.532935 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.540715 (+ 7780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.541008 (+ 293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 18:33:49.541015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 18:33:49.541021 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 18:33:49.541054 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 18:33:49.553884 (+ 12830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.576178 (+ 22294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.576184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.576188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.576338 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 18:33:49.578540 (+ 2202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 18:33:49.578547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 18:33:49.578550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 18:33:49.578877 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 18:33:49.579047 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.582545 (+ 3498us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.582769 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14653,"thread_start_us":13449,"threads_started":1}
I20260807 18:33:49.589093 19790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca: Bootstrap starting.
I20260807 18:33:49.592150 19844 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [term 1 FOLLOWER]: Refusing update from remote peer bd905673f48b4d17a56be10d35d83105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:33:49.593084 19790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca: Neither blocks nor log segments found. Creating new log.
I20260807 18:33:49.593166 19907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:33:49.594498 19790 log.cc:824] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca: Log is configured to *not* fsync() on all Append() calls
I20260807 18:33:49.586227 19641 master_runner.cc:186] Error getting master registration for 127.27.1.253:42877: OK, instance_id { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" instance_seqno: 1786127628780896 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 18:33:49.600168 19641 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 18:33:49.612344 19790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca: No bootstrap required, opened a new log
I20260807 18:33:49.614061 19921 mvcc.cc:205] Tried to move back new op lower bound from 7315978770797125632 to 7315978770322505728. Current Snapshot: MvccSnapshot[applied={T|T < 7315978770797125632}]
I20260807 18:33:49.614436 19881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd905673f48b4d17a56be10d35d83105. Latest consensus state: current_term: 1 leader_uuid: "bd905673f48b4d17a56be10d35d83105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } } }
I20260807 18:33:49.614729 19881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:49.614677 19897 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd905673f48b4d17a56be10d35d83105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } } }
I20260807 18:33:49.615002 19897 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:49.617755 19914 catalog_manager.cc:1347] Generated new cluster ID: 838aa87b4ef345069b204fabe3911f89
I20260807 18:33:49.618036 19914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:33:49.627797 19790 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } }
I20260807 18:33:49.628326 19790 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b16cc8a2a94429acee5f96c0ba4bca, State: Initialized, Role: FOLLOWER
I20260807 18:33:49.629047 19790 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } }
I20260807 18:33:49.639650 19927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } } }
I20260807 18:33:49.644639 19881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd905673f48b4d17a56be10d35d83105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } } }
I20260807 18:33:49.644898 19881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:49.641333 19790 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:33:49.641364 19763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd905673f48b4d17a56be10d35d83105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca"
I20260807 18:33:49.642025 19762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:33:49.646831 19927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:49.650005 19762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [term 1 FOLLOWER]: Refusing update from remote peer bd905673f48b4d17a56be10d35d83105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 18:33:49.651212 19907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:33:49.656472 19763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd905673f48b4d17a56be10d35d83105 in term 1.
I20260807 18:33:49.697224 19927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd905673f48b4d17a56be10d35d83105. Latest consensus state: current_term: 1 leader_uuid: "bd905673f48b4d17a56be10d35d83105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } } }
I20260807 18:33:49.697283 19933 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd905673f48b4d17a56be10d35d83105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } } }
I20260807 18:33:49.697625 19927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:49.697671 19933 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca [sys.catalog]: This master's current role is: FOLLOWER
I20260807 18:33:49.706760 19907 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd905673f48b4d17a56be10d35d83105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd905673f48b4d17a56be10d35d83105" member_type: VOTER last_known_addr { host: "127.27.1.254" port: 45751 } } peers { permanent_uuid: "f1b16cc8a2a94429acee5f96c0ba4bca" member_type: VOTER last_known_addr { host: "127.27.1.253" port: 42877 } } peers { permanent_uuid: "b492e06edd7941d682d38ef97a8ccd49" member_type: VOTER last_known_addr { host: "127.27.1.252" port: 36793 } } }
I20260807 18:33:49.707022 19907 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105 [sys.catalog]: This master's current role is: LEADER
I20260807 18:33:49.709044 19914 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:33:49.710953 19914 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:33:49.734337 19914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd905673f48b4d17a56be10d35d83105: Generated new TSK 0
I20260807 18:33:49.735948 19914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:33:49.744459 19944 catalog_manager.cc:1259] Loaded cluster ID: 838aa87b4ef345069b204fabe3911f89
I20260807 18:33:49.744658 19944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca: loading cluster ID for follower catalog manager: success
I20260807 18:33:49.748485 19944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca: acquiring CA information for follower catalog manager: success
I20260807 18:33:49.750957 19944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f1b16cc8a2a94429acee5f96c0ba4bca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 18:33:49.886049 19878 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:49.869892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.27.1.254:45751)
0807 18:33:49.870189 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.870199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.877984 (+ 7785us) server_negotiation.cc:407] Connection header received
0807 18:33:49.883566 (+ 5582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.883574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.883692 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.883833 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.885702 (+ 1869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43344: BlockingRecv error: recv got EOF from 127.0.0.1:43344 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260807 18:33:49.889408 19915 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:49.887097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.27.1.252:36793)
0807 18:33:49.887324 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.887331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.887616 (+ 285us) server_negotiation.cc:407] Connection header received
0807 18:33:49.887864 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.887870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.887979 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.888057 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.889147 (+ 1090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40476: BlockingRecv error: recv got EOF from 127.0.0.1:40476 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260807 18:33:49.889425 19879 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:49.878314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.27.1.253:42877)
0807 18:33:49.878539 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.878546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.886164 (+ 7618us) server_negotiation.cc:407] Connection header received
0807 18:33:49.887769 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.887777 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.887891 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.887974 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.889179 (+ 1205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56854: BlockingRecv error: recv got EOF from 127.0.0.1:56854 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 18:33:49.944490 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:49.928116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56864 (local address 127.27.1.253:42877)
0807 18:33:49.928332 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.928339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.935335 (+ 6996us) server_negotiation.cc:407] Connection header received
0807 18:33:49.935909 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.935917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.936183 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.936331 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:49.937238 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.938532 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.941004 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:49.941825 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:49.941942 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:49.942741 (+ 799us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:49.943538 (+ 797us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:49.943674 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:49.943989 (+ 315us) server_negotiation.cc:299] Negotiation successful
0807 18:33:49.944090 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260807 18:33:49.990955 19915 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:49.981285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.27.1.252:36793)
0807 18:33:49.981583 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:49.981591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:49.988575 (+ 6984us) server_negotiation.cc:407] Connection header received
0807 18:33:49.989160 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:49.989167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:49.989309 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:49.989406 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:49.990609 (+ 1203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40490: BlockingRecv error: recv got EOF from 127.0.0.1:40490 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20260807 18:33:50.041499 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.026213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.27.1.253:42877)
0807 18:33:50.026433 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.026442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.033434 (+ 6992us) server_negotiation.cc:407] Connection header received
0807 18:33:50.034006 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.034013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.034116 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.034198 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.034957 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.036271 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.038768 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.039622 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.039751 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.040482 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.040646 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.040717 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.041001 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.041098 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 18:33:50.093487 19915 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:50.078845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.27.1.252:36793)
0807 18:33:50.079071 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.079078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.091747 (+ 12669us) server_negotiation.cc:407] Connection header received
0807 18:33:50.091958 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.091964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.092065 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.092185 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:50.093235 (+ 1050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40504: BlockingRecv error: recv got EOF from 127.0.0.1:40504 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 18:33:50.112587 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.105439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.27.1.253:42877)
0807 18:33:50.105680 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.105686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.106200 (+ 514us) server_negotiation.cc:407] Connection header received
0807 18:33:50.106414 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.106420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.106519 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.106624 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.107155 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.108347 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.110465 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.111155 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.111272 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.111764 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.111912 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.112014 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.112180 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.112266 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 18:33:50.122143 19915 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:50.119565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.27.1.252:36793)
0807 18:33:50.119737 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.119743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.120635 (+ 892us) server_negotiation.cc:407] Connection header received
0807 18:33:50.120805 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.120812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.120911 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.121021 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:50.121886 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40510: BlockingRecv error: recv got EOF from 127.0.0.1:40510 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 18:33:50.140266 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.133029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.27.1.253:42877)
0807 18:33:50.133253 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.133258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.133829 (+ 571us) server_negotiation.cc:407] Connection header received
0807 18:33:50.134014 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.134019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.134107 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.134208 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.134691 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.135959 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.138102 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.138801 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.138936 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.139444 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.139605 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.139694 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.139884 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.139974 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260807 18:33:50.150372 19915 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:50.147439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.27.1.252:36793)
0807 18:33:50.147695 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.147703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.148715 (+ 1012us) server_negotiation.cc:407] Connection header received
0807 18:33:50.148910 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.148918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.149048 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.149159 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:50.150131 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40518: BlockingRecv error: recv got EOF from 127.0.0.1:40518 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260807 18:33:50.168403 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.161651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.27.1.253:42877)
0807 18:33:50.162395 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.162404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.162549 (+ 145us) server_negotiation.cc:407] Connection header received
0807 18:33:50.162825 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.162831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.162925 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.163029 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.163462 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.164782 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.166075 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.166905 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.167034 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.167526 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.167666 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.167767 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.167934 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.168028 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519}
W20260807 18:33:50.176726 19915 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:50.174751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.27.1.252:36793)
0807 18:33:50.174953 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.174960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.175367 (+ 407us) server_negotiation.cc:407] Connection header received
0807 18:33:50.175457 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.175463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.175558 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.175672 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:50.176434 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40526: BlockingRecv error: recv got EOF from 127.0.0.1:40526 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 18:33:50.202983 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.196263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.27.1.253:42877)
0807 18:33:50.196535 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.196541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.196927 (+ 386us) server_negotiation.cc:407] Connection header received
0807 18:33:50.197327 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.197332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.197403 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.197501 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.198244 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.198980 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.201196 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.201676 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.201774 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.202298 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.202397 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.202467 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.202640 (+ 173us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.202702 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260807 18:33:50.212571 19915 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:50.210149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.27.1.252:36793)
0807 18:33:50.210389 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.210396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.210985 (+ 589us) server_negotiation.cc:407] Connection header received
0807 18:33:50.211214 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.211218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.211295 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.211381 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:50.212346 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40538: BlockingRecv error: recv got EOF from 127.0.0.1:40538 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 18:33:50.255666 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.248897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.27.1.253:42877)
0807 18:33:50.249204 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.249211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.249643 (+ 432us) server_negotiation.cc:407] Connection header received
0807 18:33:50.249835 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.249842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.249940 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.250087 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.250501 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.251897 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.253252 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.254008 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.254191 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.254729 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.254916 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.255024 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.255199 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.255295 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260807 18:33:50.264950 19915 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:50.262966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.27.1.252:36793)
0807 18:33:50.263278 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.263285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.263548 (+ 263us) server_negotiation.cc:407] Connection header received
0807 18:33:50.263680 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.263686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.263804 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.263927 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:50.264672 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40552: BlockingRecv error: recv got EOF from 127.0.0.1:40552 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 18:33:50.344177 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.333119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.27.1.253:42877)
0807 18:33:50.333343 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.333351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.337533 (+ 4182us) server_negotiation.cc:407] Connection header received
0807 18:33:50.337876 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.337882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.337969 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.338063 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.338540 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.339842 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.341859 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.342593 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.342737 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.343288 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.343453 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.343541 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.343725 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.343826 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260807 18:33:50.359549 19915 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:33:50.356544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.27.1.252:36793)
0807 18:33:50.356714 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.356719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.358098 (+ 1379us) server_negotiation.cc:407] Connection header received
0807 18:33:50.358317 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.358323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.358399 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.358524 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 18:33:50.359325 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40566: BlockingRecv error: recv got EOF from 127.0.0.1:40566 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 18:33:50.432367 19894 catalog_manager.cc:1259] Loaded cluster ID: 838aa87b4ef345069b204fabe3911f89
I20260807 18:33:50.432518 19894 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49: loading cluster ID for follower catalog manager: success
I20260807 18:33:50.435330 19894 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49: acquiring CA information for follower catalog manager: success
I20260807 18:33:50.437461 19894 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b492e06edd7941d682d38ef97a8ccd49: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 18:33:50.496417 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.491494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.27.1.253:42877)
0807 18:33:50.491693 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.491698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.492031 (+ 333us) server_negotiation.cc:407] Connection header received
0807 18:33:50.492201 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.492204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.492276 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.492367 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.492754 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.493459 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.494773 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.495283 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.495378 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.495802 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.495901 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.495992 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.496143 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.496208 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 18:33:50.507730 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.502369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.27.1.252:36793)
0807 18:33:50.502601 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.502608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.502794 (+ 186us) server_negotiation.cc:407] Connection header received
0807 18:33:50.502952 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.502956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.503106 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.503265 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.503610 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.504346 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.505612 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.506034 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.506126 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.506569 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.507095 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.507268 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.507333 (+ 65us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.507396 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 18:33:50.525807 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.513590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.27.1.253:42877)
0807 18:33:50.513806 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.513814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.518920 (+ 5106us) server_negotiation.cc:407] Connection header received
0807 18:33:50.519252 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.519258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.519349 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.519418 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.520523 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.521442 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.523689 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.524276 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.524394 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.525012 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.525143 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.525208 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.525437 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.525519 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 18:33:50.526054 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.519231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40584 (local address 127.27.1.252:36793)
0807 18:33:50.519423 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.519429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.520375 (+ 946us) server_negotiation.cc:407] Connection header received
0807 18:33:50.520674 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.520679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.520757 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.520846 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.521408 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.522181 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.524363 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.524807 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.524944 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.525347 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.525455 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.525555 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.525686 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.525759 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 18:33:50.527019 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 18:33:50.513418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.27.1.254:45751)
0807 18:33:50.519321 (+ 5903us) server_negotiation.cc:206] Beginning negotiation
0807 18:33:50.519328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 18:33:50.519346 (+ 18us) server_negotiation.cc:407] Connection header received
0807 18:33:50.519482 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 18:33:50.519487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 18:33:50.519723 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 18:33:50.519994 (+ 271us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 18:33:50.520798 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.521970 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.524215 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 18:33:50.524835 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 18:33:50.525455 (+ 620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 18:33:50.525640 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 18:33:50.526092 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 18:33:50.526690 (+ 598us) server_negotiation.cc:1035] Waiting for connection context
0807 18:33:50.526741 (+ 51us) server_negotiation.cc:299] Negotiation successful
0807 18:33:50.526816 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5784,"thread_start_us":5652,"threads_started":1}
I20260807 18:33:50.531591 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 19641
I20260807 18:33:50.532660 19707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:50.646483 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 19641
I20260807 18:33:50.671108 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 19720
I20260807 18:33:50.672273 19785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:50.781387 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 19720
I20260807 18:33:50.805120 27655 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu pid 19798
I20260807 18:33:50.806316 19865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 18:33:50.916390 27655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrFJVJ0/build/asan/bin/kudu with pid 19798
2026-08-07T18:33:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4579 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9323 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (375859 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST